Environment evaluation report table character recognition and extraction method and device

Abstract

The invention discloses an environment evaluation report table character recognition and extraction method and device, and the method comprises the steps: building a table recognition neural network model based on an environment evaluation report, and carrying out the training verification of the table recognition neural network model through a first training set and a first verification set; establishing a character recognition neural network model based on the environment evaluation report, and performing training verification on the character recognition neural network model by using the second training set and the second verification set; and carrying out character recognition on each cell in the environment evaluation report by utilizing the character recognition neural network model after training verification, and carrying out structured output and storage on recognition content of a character recognition result by utilizing table positioning and table row and column positioning information in the environment evaluation report and the character recognition result of each cell. According to the method, the table can be recognized and extracted from the environment evaluation report, table characters are extracted, recognized and structurally output according to table cells, and a foundation is laid for machine-assisted verification by using a large model.

Description

Technical Field

[0001] The present invention belongs to the field of environmental impact assessment, neural network, deep learning, and OCR recognition technology, and specifically relates to a method and device for recognizing and extracting text in an environmental assessment report form. Background Art

[0002] An environmental impact assessment report, whose full name is environmental impact report, also known as an environmental evaluation report, is an environmental impact assessment document prepared by an organization with relevant environmental impact assessment qualifications for construction projects that may cause major environmental impacts, which comprehensively evaluates the environmental impacts generated.

[0003] At present, when the environmental protection department reviews the project environmental impact report, the report submitted by the project environmental assessment unit contains a large number of pollutant emission data tables and other environmental impact data tables depending on the project type. Usually, a scanned copy of an environmental impact report has hundreds or even thousands of pages, of which the table data is usually more than a hundred. These pages require professionals to verify them in turn according to the submitted content, combined with specific national standards and industry standards. On the one hand, it increases the workload of environmental assessment staff and reduces work efficiency. On the other hand, it also affects the timeliness of project approval, and thus affects the progress of project implementation. The use of large models for machine-assisted verification can quickly and accurately review project environmental impact reports, but structured table recognition content is required before the large model is machine-assisted verification. For scanned copies of environmental impact reports with hundreds or even thousands of pages, how to efficiently perform text recognition and extraction has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] To this end, the present invention provides a method and device for recognizing and extracting text from an environmental assessment report table, which can recognize and extract tables from an environmental assessment report, and extract and recognize table text according to table cells, and then perform structured output of the extracted and recognized content, so as to lay the foundation for further using a large model for machine-assisted verification.

[0005] In order to achieve the above object, the present invention provides the following technical solution: a method for recognizing and extracting text in an environmental assessment report form, comprising the following steps: According to the preset industry classification standard, obtain the report data sample of the environmental assessment data table in the scanned copy of the environmental assessment report of the specified industry, and use the report data sample to construct the image data set of the environmental assessment data table of the specified industry; Performing data preprocessing on the image data set of the environmental evaluation data table of the specified industry, and dividing the image data set of the environmental evaluation data table after the data preprocessing into a first training set and a first validation set; Establishing a table recognition neural network model based on the environmental assessment report, training the table recognition neural network model using the first training set, verifying the table recognition neural network model using the first verification set, and generating the table recognition neural network model that has been trained and verified; Using the trained table recognition neural network model to recognize the scanned copy of the environmental assessment report, and obtaining the table location and table row and column location in the environmental assessment report; According to the table positioning and table row and column positioning information in the environmental assessment report, image segmentation is performed on the table cells in the environmental assessment report to obtain a segmented table cell image in the environmental assessment report; Using the segmented table cell images in the environmental assessment report, a picture data set of table cell images in the environmental assessment report is established, and the picture data set of table cell images is divided into a second training set and a second validation set; Establishing a character recognition neural network model based on the environmental assessment report, training the character recognition neural network model using the second training set, verifying the character recognition neural network model using the second verification set, and generating the trained and verified character recognition neural network model; Using the trained and verified character recognition neural network model, perform text recognition on each cell in the environmental assessment report to obtain a text recognition result for each cell, and perform data correction and cleaning on the text recognition result for each cell; By using the table positioning and table row and column positioning information in the environmental assessment report and the text recognition results of each cell, the recognition content of the text recognition results is output and stored in a structured manner.

[0006] As a preferred solution for the method of text recognition and extraction of environmental assessment report forms, data preprocessing of the image data set of the environmental assessment data form of the specified industry includes grayscale normalization processing; The formula for grayscale normalization is: ; In the formula, is the gray value of a single image pixel after normalization, is the gray value of a single image pixel to be processed in the original image, is the minimum gray value of the image to be processed, is the maximum grayscale value of the image to be processed, is the maximum value of the image interval to be processed, The minimum value of the image interval to be processed.

[0007] As a preferred solution for the method of text recognition and extraction of environmental assessment report forms, the data preprocessing of the image data set of the environmental assessment data table of the specified industry also includes data enhancement processing; the data enhancement processing includes performing at least one of random scaling, center cropping, random horizontal flipping, image denoising, binarization, and edge detection operations on the images in the image data set of the environmental assessment data table of the specified industry.

[0008] As a preferred solution for the method of recognizing and extracting text from the table of the environmental assessment report, in the process of establishing a table recognition neural network model based on the environmental assessment report, the table recognition neural network model adopts a hybrid model of the fast convolutional neural network of the deformable CNN and the feature pyramid network. The expression formula of the hybrid model of the fast convolutional neural network of the deformable CNN and the feature pyramid network is: ; In the formula, * is the convolution operation, F is the filter, I For images, K Defined as 1 / 2 of the filter size, H is the image height, W is the image width, i , j To define where to perform the convolution, Defined as the vertical offset, Defined as a horizontal offset.

[0009] As a preferred solution for the text recognition and extraction method of the environmental assessment report form, in the process of establishing a character recognition neural network model based on the environmental assessment report, the character recognition neural network model uses a deep residual network model as a feature extractor, and the character recognition neural network model is batch normalized. The formula for batch normalization of the character recognition neural network model is: ; In the formula, is the output value of batch normalization, is the input value of batch normalization, is the mean of all samples in a specified batch, is the variance of all samples in the specified batch, and ϵ is a constant set to avoid division by zero; After batch normalization of the character recognition neural network model, the method further includes performing linear transformation and translation on the batch normalization results: ; In the formula, is the output value after linear transformation and translation, and γ and β are both learnable parameters.

[0010] As a preferred solution for the method of recognizing and extracting text in the environmental assessment report form, the process of establishing a character recognition neural network model based on the environmental assessment report also includes performing Dropout regularization on the character recognition neural network model: y=mask * x ; Where y is the model output parameter value; x Input parameter values ​​for the model. Mask is a vector of Boolean values ​​0 and 1, indicating whether each neuron is "pruned"; mask = Bernoulli(p,n); Where, Bernoulli is the discrete probability distribution function of a binary random variable, p is the probability, n is the number of neuron layers, and Bernoulli(p,n) means sampling a vector of size n with probability p.

[0011] As a preferred method for identifying and extracting text from environmental assessment report tables, in the process of obtaining table positioning and table row and column positioning in the environmental assessment report: Get the relative coordinate information of cells in the table, organize the relative coordinate information of cells and output it as structured data. The expression of structured data is [x0, y0, x1, y1, position]; The position of the cell is analyzed through a recursive algorithm to form position positioning information. Position is expressed as additional positioning information, which includes the row and column information of the cell and the parent cell information of the cell. The expression of the additional positioning information is [ID, H`, V, Parent, Type], where ID is the unique identifier of the cell; H` and V are the internal row and column information of the cell; Parent identifies the ID of the parent cell to which the cell belongs. If there is nested cell, Parent is used to specify the ID of the parent cell to which the cell belongs; Type is marked as 1, indicating that the cell is a metadata cell; Type is marked as 0, indicating that the cell has an embedded cell; Type is marked as 2, indicating that the cell spans rows and columns; Type is marked as -1, indicating that it is header data.

[0012] The present invention also provides a device for identifying and extracting text in an environmental assessment report form, comprising: A table image data set construction module is used to obtain report data samples of environmental assessment data tables in scanned copies of environmental assessment reports for a specified industry according to preset industry classification standards, and use the report data samples to construct an image data set of environmental assessment data tables for the specified industry; A table image data set processing module is used to perform data preprocessing on the image data set of the environmental evaluation data table of the specified industry, and divide the image data set of the environmental evaluation data table after data preprocessing into a first training set and a first verification set; A table recognition neural network model building module is used to establish a table recognition neural network model based on the environmental assessment report, train the table recognition neural network model using the first training set, verify the table recognition neural network model using the first verification set, and generate the table recognition neural network model that has been trained and verified; A positioning information acquisition module is used to use the trained table recognition neural network model to recognize the scanned copy of the environmental assessment report and obtain the table location and table row and column location in the environmental assessment report; The cell image processing module is used to perform image segmentation on the table cells in the environmental assessment report according to the table positioning and table row and column positioning information in the environmental assessment report, and obtain the segmented table cell images in the environmental assessment report; A cell image data set construction module is used to use the segmented table cell images in the environmental assessment report to establish an image data set of table cell images in the environmental assessment report, and divide the image data set of table cell images into a second training set and a second validation set; A character recognition neural network model building module, used to establish a character recognition neural network model based on the environmental assessment report, train the character recognition neural network model using the second training set, verify the character recognition neural network model using the second verification set, and generate the trained and verified character recognition neural network model; A cell text recognition processing module is used to use the trained and verified character recognition neural network model to perform text recognition on each cell in the environmental assessment report, obtain the text recognition result of each cell, and perform data correction and cleaning on the text recognition result of each cell; The cell text recognition result output module is used to utilize the table positioning and table row and column positioning information in the environmental assessment report, as well as the text recognition result of each cell, to output and store the recognition content of the text recognition result in a structured manner.

[0013] As a preferred solution of the device for recognizing and extracting text from the environmental assessment report form, in the table image data set processing module, the data preprocessing of the image data set of the environmental assessment data table of the specified industry includes grayscale normalization processing; The formula for grayscale normalization is: ; In the formula, is the gray value of a single image pixel after normalization, is the gray value of a single image pixel to be processed in the original image, is the minimum gray value of the image to be processed, is the maximum grayscale value of the image to be processed, is the maximum value of the image interval to be processed, is the minimum value of the image interval to be processed; In the table image data set processing module, the data preprocessing of the image data set of the environmental assessment data table of the specified industry also includes data enhancement processing; the data enhancement processing includes performing at least one of random scaling, center cropping, random horizontal flipping, image denoising, binarization, and edge detection operations on the images in the image data set of the environmental assessment data table of the specified industry.

[0014] As a preferred solution of the device for recognizing and extracting text from environmental assessment report tables, in the table recognition neural network model construction module: The table recognition neural network model adopts a hybrid model of a fast convolutional neural network of a deformable CNN and a feature pyramid network. The expression formula of the hybrid model of a fast convolutional neural network of a deformable CNN and a feature pyramid network is: ; In the formula, * is the convolution operation, F is the filter, I For images, K Defined as 1 / 2 of the filter size, H is the image height, W is the image width, i , j To define where to perform the convolution, Defined as the vertical offset, Defined as a horizontal offset.

[0015] As a preferred solution of the device for recognizing and extracting characters from environmental assessment report forms, in the character recognition neural network model building module: The character recognition neural network model uses a deep residual network model as a feature extractor, and performs batch normalization on the character recognition neural network model. The formula for performing batch normalization on the character recognition neural network model is: ; In the formula, is the output value of batch normalization, is the input value of batch normalization, is the mean of all samples in a specified batch, is the variance of all samples in the specified batch, and ϵ is a constant set to avoid division by zero; In the character recognition neural network model building module, the batch normalized results are linearly transformed and translated: ; In the formula, is the output value after linear transformation and translation, γ and β are both learnable parameters; The character recognition neural network model construction module also includes performing Dropout regularization on the character recognition neural network model: y=mask * x ; Where y is the model output parameter value; x Input parameter values ​​for the model. Mask is a vector of Boolean values ​​0 and 1, indicating whether each neuron is "pruned"; mask = Bernoulli(p,n); Where, Bernoulli is the discrete probability distribution function of a binary random variable, p is the probability, n is the number of neuron layers, and Bernoulli(p,n) means sampling a vector of size n with probability p.

[0016] As a preferred solution of the device for recognizing and extracting text in the environmental assessment report form, in the cell text recognition processing module: Get the relative coordinate information of cells in the table, organize the relative coordinate information of cells and output it as structured data. The expression of structured data is [x0, y0, x1, y1, position]; The position of the cell is analyzed through a recursive algorithm to form position positioning information. Position is expressed as additional positioning information, which includes the row and column information of the cell and the parent cell information of the cell. The expression of the additional positioning information is [ID, H`, V, Parent, Type], where ID is the unique identifier of the cell; H` and V are the internal row and column information of the cell; Parent identifies the ID of the parent cell to which the cell belongs. If there is nested cell, Parent is used to specify the ID of the parent cell to which the cell belongs; Type is marked as 1, indicating that the cell is a metadata cell; Type is marked as 0, indicating that the cell has an embedded cell; Type is marked as 2, indicating that the cell spans rows and columns; Type is marked as -1, indicating that it is header data.

[0017] The present invention has the following advantages: the present invention can accurately identify environmental assessment reports, can identify header text and cell position layout information and specific text information in the cells, and form structured data output through data cleaning and organization, laying the foundation for further verification; it can locate the position information of each cell in the grid; the table recognition result is used as the input of the lower-level text character recognition, and the text character recognition is based on the residual neural network, and has a higher accuracy rate than the traditional OCR through model optimization and targeted training and learning, and is particularly suitable for specific scenes and special characters in environmental protection reports; the present invention can realize continuous recognition and automatic synthesis of cross-page tables, can output recognition results in a structured manner, and can output them in formats such as CSV, Excel or database, greatly improving the work efficiency of verification personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0019] Figure 1 This is a sample diagram of table text used for training and testing in the embodiment of the present invention; in order to prevent information leakage, the sensitive data is blurred; Figure 2 A flow chart of a method for identifying and extracting text from an environmental assessment report form provided in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a fast convolutional neural network with a deformable CNN in the method for recognizing and extracting text in an environmental assessment report form provided in an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a residual neural network for character recognition in a method for recognizing and extracting text in an environmental assessment report form provided in an embodiment of the present invention; Figure 5 A schematic diagram of a residual unit structure in a residual neural network structure for character recognition provided in an embodiment of the present invention; Figure 6 This is an architecture diagram of the device for recognizing and extracting text from an environmental assessment report form provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] See also Figure 1 , is a scanned image of an example environmental assessment report. It can be clearly seen that the image contains a variety of data tables. Some of these tables are on the same page, and some tables span pages. The table structure includes a title and a table body. The table body is divided into multiple basic cells. Some basic cells can be further divided into smaller cells. The content of the cells includes Chinese, English, numbers and various symbols. In order to be able to identify and extract tables from environmental assessment reports, and extract and identify table text according to table cells, and then perform structured output on the extracted and identified content, it lays the foundation for further using large models for machine-assisted verification. The present invention provides a method and device for recognizing and extracting text from environmental assessment report tables. The following is the specific content of an embodiment of the present invention.

[0022] Example 1 See also Figure 2 The embodiment of the present invention provides a method for recognizing and extracting text in an environmental assessment report form, comprising the following steps: S1. According to the preset industry classification standard, obtain the report data sample of the environmental assessment data table in the scanned copy of the environmental assessment report of the specified industry, and use the report data sample to construct the image data set of the environmental assessment data table of the specified industry; S2, performing data preprocessing on the image data set of the environmental assessment data table of the specified industry, and dividing the image data set of the environmental assessment data table after the data preprocessing into a first training set and a first validation set; S3, establishing a table recognition neural network model based on the environmental assessment report, training the table recognition neural network model using the first training set, verifying the table recognition neural network model using the first verification set, and generating the table recognition neural network model that has been trained and verified; S4, using the trained table recognition neural network model to recognize the scanned copy of the environmental assessment report, and obtaining the table location and table row and column location in the environmental assessment report; S5. According to the table positioning and table row and column positioning information in the environmental assessment report, image segmentation is performed on the table cells in the environmental assessment report to obtain segmented table cell images in the environmental assessment report; S6, using the segmented table cell images in the environmental assessment report to establish a picture data set of table cell images in the environmental assessment report, and dividing the picture data set of table cell images into a second training set and a second validation set; S7, establishing a character recognition neural network model based on the environmental assessment report, training the character recognition neural network model using the second training set, verifying the character recognition neural network model using the second verification set, and generating the trained and verified character recognition neural network model; S8, using the trained and verified character recognition neural network model, perform text recognition on each cell in the environmental assessment report to obtain a text recognition result for each cell, and perform data correction and cleaning on the text recognition result for each cell; S9. Using the table positioning and table row and column positioning information in the environmental assessment report, as well as the text recognition results of each cell, the recognition content of the text recognition results is output and stored in a structured manner.

[0023] In this embodiment, in step S1, when obtaining report data samples of environmental assessment data tables in scanned copies of environmental assessment reports for a specified industry, public data sets such as ICDAR-3, ICDAR-7, MORMOT, and WTW data sets can be used. At the same time, the table images contained in the existing scanned copies of environmental impact reports are obtained; the obtained table images are integrated and mixed with the public data sets to form a high-coverage image data set containing table images.

[0024] In this embodiment, in step S2, the data preprocessing performed on the image data set of the environmental assessment data table of the specified industry includes grayscale normalization processing; The formula for grayscale normalization is: ; In the formula, is the gray value of a single image pixel after normalization, is the gray value of a single image pixel to be processed in the original image, is the minimum gray value of the image to be processed, is the maximum grayscale value of the image to be processed, is the maximum value of the image interval to be processed, The minimum value of the image interval to be processed.

[0025] Specifically, the grayscale value of each pixel in the original image is subtracted from the minimum grayscale value, and the result is divided by the grayscale value range (that is, the maximum value minus the minimum value), multiplied by the range size of the specified interval, and finally added to the minimum value of the specified interval to enhance the contrast and details of the image. In this example, the interval selected is [0, 1].

[0026] In this embodiment, in step S2, the data preprocessing of the image data set of the environmental evaluation data table of the specified industry also includes data enhancement processing; the data enhancement processing includes random scaling, center cropping, random horizontal flipping, image denoising, binarization, and edge detection operations on the images in the image data set of the environmental evaluation data table of the specified industry. Among them, random scaling is to randomly scale the image in the interval [0.8, 1.0]. Center cropping is to enlarge the image in equal proportion to the smaller side, and cut the image of the original image size based on the center of the enlarged image. Random horizontal flipping is to randomly flip the original image horizontally to generate a new training set. Image denoising, binarization, and edge detection operations have related technologies themselves, which will not be repeated here.

[0027] In this embodiment, in step S3, in the process of establishing a table recognition neural network model based on the environmental assessment report, the table recognition neural network model adopts a hybrid model of a fast convolutional neural network of a deformable CNN and a feature pyramid network. Figure 3 The hybrid model of the fast convolutional neural network and feature pyramid network of the deformable CNN includes an input layer, a deep deformable CNN layer (Deep Deformable CNN Backbone), an input feature map layer (Feature Maps), a region proposal network layer (Region Proposel Network), a deformable ROI pooling layer (Defromable ROIpooling), a pre-classification layer (Per-region Classifier) ​​and an output layer.

[0028] Among them, the fast convolutional neural network of the deformable CNN is an automatic feature extractor with the ability to automatically discover features that are useful for the task at hand. This automatic extraction of features is based on a hierarchy of layers, where the initial layers extract primitive features such as edges and gradients, while the layers at the top of the hierarchy extract very abstract features such as complete objects or some of its salient parts. This traversal in the hierarchy leads to an increase in the effective receptive field of a particular neuron in the original input image. In a given convolutional layer, the effective receptive field of all neurons is the same. This property is problematic for layers at the top of the hierarchy, where different objects may appear at arbitrary scales and with arbitrary transformations. The presence of these transformations requires the ability to dynamically adapt the receptive field of neurons based on their inputs. Therefore, the fast convolutional neural network model is equipped with a deformable CNN instead of a traditional CNN, whose neurons are not limited to a predefined receptive field. Each neuron can change its receptive field by generating explicit offsets based on the input, which themselves depend on the previous feature map. This allows the convolutional layer filters to adapt to different scales and transformations by adjusting the receptive field of the input itself. This deformable convolutional layer adds a set of convolutional layers to generate filter offsets for each position in the image. Since tables can be of arbitrary scales and arbitrarily transformed (in orientation, etc.), deformable convolution operations play a key role in the table detection task.

[0029] Among them, the expression formula of the hybrid model of the fast convolutional neural network and feature pyramid network of the deformable CNN is: ; In the formula, * is the convolution operation, F is the filter, I For images, K Defined as 1 / 2 of the filter size, H is the image height, W is the image width, i , j To define where to perform the convolution, Defined as the vertical offset, Defined as the horizontal offset. The vertical and horizontal offsets are generated by another convolutional block, consuming the feature map produced by the previous layer. The offset can be fractional and is implemented by bilinear interpolation.

[0030] To train a hybrid model of Fast Convolutional Neural Network and Feature Pyramid Network for Deformable CNN, three different anchor ratios (0.5, 1, and 2) and five different anchor scales (2, 4, 8, 16, and 32) were used. For the first 200 iterations, the optimized model was optimized with an initial learning rate of 0.000125. A learning rate of 0.00125 was then used with a learning rate decay step size at 4, 16, and 32 cycles. The model was optimized for 50 epochs. The maximum image size was limited to 1280×800. Images larger than this size were resized to keep the aspect ratio unchanged.

[0031] In this embodiment, in step S7, in the process of establishing a character recognition neural network model based on the environmental assessment report, a residual neural network model is selected. The residual neural network structure is as follows: Figure 4 As shown in the figure, it includes input layer, convolutional layer, residual block (ResNet Block), global average pooling layer (Global AveragePooling Layer), fully connected layer (Fully Connected Layer) and output layer.

[0032] The convolution layer consists of two convolution layers and a pooling layer. A 3×3 convolution kernel is used for transformation to extract image features. The feature extraction results are then subjected to two 3x3 convolution layers to obtain the output results. The global average pooling layer uses the average value of each feature map as the representation of the feature, which improves the representation capability while ensuring that it does not overfit. The fully connected layer is used for feature classification output results, and the ReLU activation function is used to perform linear and nonlinear transformations on the feature vectors of neurons.

[0033] Among them, the residual unit in the residual neural network structure is as follows Figure 5 As shown in the figure, two consecutive convolutional layers are followed by batch normalization and ReLU activation functions. The activation function can enhance the nonlinear expression ability of the network and alleviate the problem of gradient disappearance. The purpose of Addition is to add the outputs of the two parts before and after the network. First, a 1×1 convolutional layer is used to make the dimensions and number of channels of the two parts the same, and then they are added to get the output. Finally, the ReLU function is used for activation again. The character recognition neural network model is trained using the preprocessed data set, and the predicted results of the character recognition neural network model are compared with the real label data through the cross entropy loss function to calculate the distance between the two. The loss value decreases with the increase of the number of training samples. When it decreases to below 1e-3, it means that the character recognition neural network model has sufficient recognition ability and accuracy, and the training step is completed.

[0034] In this embodiment, in step S7, in the process of establishing a character recognition neural network model based on the environmental assessment report, the character recognition neural network model uses a deep residual network model as a feature extractor, and batch normalization is performed on the character recognition neural network model. The formula for batch normalization of the character recognition neural network model is: ; In the formula, is the output value of batch normalization, is the input value of batch normalization, is the mean of all samples in a specified batch, is the variance of all samples in the specified batch, and ϵ is a constant set to avoid division by zero; After batch normalization of the character recognition neural network model, the method further includes performing linear transformation and translation on the batch normalization results: ; In the formula, is the output value after linear transformation and translation, and γ and β are both learnable parameters.

[0035] In this embodiment, in step S7, the process of establishing a character recognition neural network model based on the environmental assessment report also includes performing Dropout regularization on the character recognition neural network model: y=mask * x ; Where y is the model output parameter value; x Input parameter values ​​for the model. Mask is a vector of Boolean values ​​0 and 1, indicating whether each neuron is "pruned"; mask = Bernoulli(p,n); Where, Bernoulli is the discrete probability distribution function of a binary random variable, p is the probability, n is the number of neuron layers, and Bernoulli(p,n) means sampling a vector of size n with probability p.

[0036] In this way, the network complexity and overfitting can be reduced by randomly setting the output of some neurons to zero during training. In each round of training, some neurons are randomly selected with probability p to have zero output, thus obtaining a "pruned" model. During testing, the Dropout operation is not performed, but the output of all neurons is multiplied by p for scaling. In the model testing phase, accuracy is used as an evaluation indicator, and the accuracy test is performed on the validation data set.

[0037] In this embodiment, in step S4, in the process of obtaining the table location and table row and column location in the environmental assessment report: Get the relative coordinate information of the cells in the table, organize the relative coordinate information of the cells and output it as structured data. The expression of the structured data is [x0, y0, x1, y1, position]; (x0, y0) and (x1, y1) are the relative coordinates of the starting and ending positions of the cells, and the relative coordinates are 32-bit floating point numbers between 0 and 1.

[0038] The position of the cell is analyzed through a recursive algorithm to form position information. Position is expressed as additional positioning information, which includes the row and column information of the cell and the parent cell information of the cell. The expression of the additional positioning information is [ID, H`, V, Parent, Type], where ID is the unique identifier of the cell, and -1 is specified as the top-level cell (i.e. the entire table). H` and V are the internal row and column information of the cell. Parent identifies the ID of the parent cell to which the cell belongs. The Parent of the top-level cell is fixed to -1. If there is nested cell, Parent is used to specify the ID of the parent cell to which the cell belongs. Type is marked as 1, indicating that the cell is a metadata cell. Type is marked as 0, indicating that the cell has an embedded cell. Type is marked as 2, indicating that the cell spans rows and columns. Type is marked as -1, indicating that it is header data.

[0039] In this embodiment, in step S8, the established character recognition neural network model is used to perform text recognition on each unit segmentation image, and the data is sorted and output as structured data, such as [x0, y0, x1, y1, content, position], where (x0, y0) and (x1, y1) are the relative coordinates of the start and end positions of the cell, content is the recognized text information, and position is the additional row and column information. The formed content text information is corrected and cleaned, and spaces and other information are removed to form standard recognized text information. Then in step S9, the formed recognized text information is converted into CSV, EXCEL or database according to user needs and stored on a storage medium.

[0040] In summary, the present invention obtains report data samples of environmental assessment data tables in scanned copies of environmental assessment reports of specified industries according to preset industry classification standards, and uses the report data samples to construct image data sets of environmental assessment data tables of specified industries; performs data preprocessing on the image data sets of environmental assessment data tables of specified industries, and divides the image data sets of environmental assessment data tables after data preprocessing into a first training set and a first verification set; establishes a table recognition neural network model based on the environmental assessment report, trains the table recognition neural network model using the first training set, verifies the table recognition neural network model using the first verification set, and generates the table recognition neural network model that has been trained and verified; uses the trained table recognition neural network model to recognize scanned copies of environmental assessment reports, and obtains table positioning and table row and column positioning in the environmental assessment report; and identifies table cells in the environmental assessment report according to the table positioning and table row and column positioning information in the environmental assessment report. Perform image segmentation to obtain the segmented table cell images in the environmental assessment report; use the segmented table cell images in the environmental assessment report to establish a picture data set of the table cell images in the environmental assessment report, and divide the picture data set of the table cell images into a second training set and a second verification set; establish a character recognition neural network model based on the environmental assessment report, use the second training set to train the character recognition neural network model, use the second verification set to verify the character recognition neural network model, and generate the trained and verified character recognition neural network model; use the trained and verified character recognition neural network model to perform text recognition on each cell in the environmental assessment report to obtain the text recognition result of each cell, and perform data correction and cleaning on the text recognition result of each cell; use the table positioning and table row and column positioning information in the environmental assessment report, and the text recognition result of each cell, to structure the output and storage of the recognition content of the text recognition result. The present invention can accurately identify environmental assessment reports, can identify header text, cell position layout information and specific text information in the cells, and form structured data output through data cleaning and organization, laying the foundation for further verification; it can locate the position information of each cell in the grid; the table recognition result is used as the input of the lower-level text character recognition, and the text character recognition is based on the residual neural network. Compared with the traditional OCR, it has a higher accuracy rate through model optimization and targeted training and learning, and is particularly suitable for specific scenes and special characters in environmental protection reports; the present invention can realize continuous recognition and automatic synthesis of cross-page tables, can output recognition results in a structured manner, and can output them in formats such as CSV, Excel or database, greatly improving the work efficiency of verification personnel.

[0041] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0042] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0043] Example 2 See also Figure 6 The embodiment of the present invention also provides a device for identifying and extracting text in an environmental assessment report form, comprising: The table image data set construction module 100 is used to obtain the report data sample of the environmental assessment data table in the scanned copy of the environmental assessment report of the specified industry according to the preset industry classification standard, and use the report data sample to construct the image data set of the environmental assessment data table of the specified industry; The table image data set processing module 200 is used to perform data preprocessing on the image data set of the environmental evaluation data table of the specified industry, and divide the image data set of the environmental evaluation data table after data preprocessing into a first training set and a first verification set; The table recognition neural network model building module 300 is used to establish a table recognition neural network model based on the environmental assessment report, train the table recognition neural network model using the first training set, verify the table recognition neural network model using the first verification set, and generate the table recognition neural network model that has been trained and verified; The positioning information acquisition module 400 is used to recognize the scanned copy of the environmental assessment report using the trained table recognition neural network model to obtain the table position and table row and column position in the environmental assessment report; The cell image processing module 500 is used to perform image segmentation on the table cells in the environmental assessment report according to the table positioning and table row and column positioning information in the environmental assessment report, and obtain the segmented table cell images in the environmental assessment report; The cell image data set building module 600 is used to build an image data set of the table cell images in the environmental assessment report by using the segmented table cell images in the environmental assessment report, and divide the image data set of the table cell images into a second training set and a second validation set; The character recognition neural network model building module 700 is used to establish a character recognition neural network model based on the environmental assessment report, train the character recognition neural network model using the second training set, verify the character recognition neural network model using the second verification set, and generate the character recognition neural network model that has been trained and verified; The cell text recognition processing module 800 is used to use the trained and verified character recognition neural network model to perform text recognition on each cell in the environmental assessment report, obtain the text recognition result of each cell, and perform data correction and cleaning on the text recognition result of each cell; The cell text recognition result output module 900 is used to utilize the table positioning and table row and column positioning information in the environmental assessment report, as well as the text recognition result of each cell, to output and store the recognition content of the text recognition result in a structured manner.

[0044] In this embodiment, in the table image data set processing module 200, the data preprocessing performed on the image data set of the environmental evaluation data table of the specified industry includes grayscale normalization processing; The formula for grayscale normalization is: ; In the formula, is the gray value of a single image pixel after normalization, is the gray value of a single image pixel to be processed in the original image, is the minimum gray value of the image to be processed, is the maximum grayscale value of the image to be processed, is the maximum value of the image interval to be processed, is the minimum value of the image interval to be processed; In the table image data set processing module 200, the data preprocessing of the image data set of the environmental assessment data table of the specified industry also includes data enhancement processing; the data enhancement processing includes performing at least one of random scaling, center cropping, random horizontal flipping, image denoising, binarization, and edge detection operations on the images in the image data set of the environmental assessment data table of the specified industry.

[0045] In this embodiment, in the table recognition neural network model construction module 300: The table recognition neural network model adopts a hybrid model of a fast convolutional neural network of a deformable CNN and a feature pyramid network. The expression formula of the hybrid model of a fast convolutional neural network of a deformable CNN and a feature pyramid network is: ; In the formula, * is the convolution operation, F is the filter, I For images, K Defined as 1 / 2 of the filter size, H is the image height, W is the image width, i , j To define where to perform the convolution, Defined as the vertical offset, Defined as a horizontal offset.

[0046] In this embodiment, in the character recognition neural network model construction module 700: The character recognition neural network model uses a deep residual network model as a feature extractor, and performs batch normalization on the character recognition neural network model. The formula for performing batch normalization on the character recognition neural network model is: ; In the formula, is the output value of batch normalization, is the input value of batch normalization, is the mean of all samples in a specified batch, is the variance of all samples in the specified batch, and ϵ is a constant set to avoid division by zero; In the character recognition neural network model building module 700, the batch normalized results are linearly transformed and translated: ; In the formula, is the output value after linear transformation and translation, γ and β are both learnable parameters; The character recognition neural network model construction module also includes performing Dropout regularization on the character recognition neural network model: y=mask * x ; Where y is the model output parameter value; x Input parameter values ​​for the model. Mask is a vector of Boolean values ​​0 and 1, indicating whether each neuron is "pruned"; mask = Bernoulli(p,n); Where, Bernoulli is the discrete probability distribution function of a binary random variable, p is the probability, n is the number of neuron layers, and Bernoulli(p,n) means sampling a vector of size n with probability p.

[0047] In this embodiment, in the cell text recognition processing module 800: Get the relative coordinate information of cells in the table, organize the relative coordinate information of cells and output it as structured data. The expression of structured data is [x0, y0, x1, y1, position]; The position of the cell is analyzed through a recursive algorithm to form position positioning information. Position is expressed as additional positioning information, which includes the row and column information of the cell and the parent cell information of the cell. The expression of the additional positioning information is [ID, H`, V, Parent, Type], where ID is the unique identifier of the cell; H` and V are the internal row and column information of the cell; Parent identifies the ID of the parent cell to which the cell belongs. If there is nested cell, Parent is used to specify the ID of the parent cell to which the cell belongs; Type is marked as 1, indicating that the cell is a metadata cell; Type is marked as 0, indicating that the cell has an embedded cell; Type is marked as 2, indicating that the cell spans rows and columns; Type is marked as -1, indicating that it is header data.

[0048] It should be noted that the information interaction, execution process and other contents between the modules of the above-mentioned device are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown in the previous part of the present application, and will not be repeated here.

[0049] Example 3 Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which a program code for a method for recognizing and extracting text from an environmental assessment report form is stored. The program code includes instructions for executing the method for recognizing and extracting text from an environmental assessment report form of embodiment 1 or any possible implementation thereof.

[0050] Computer-readable storage media can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0051] Example 4 Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor; The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the environmental assessment report form text recognition and extraction method of Example 1 or any possible implementation method thereof.

[0052] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor implemented by reading software codes stored in a memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0053] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0054] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0055] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for identifying and extracting text from an environmental assessment report form, characterized in that: The following steps are involved: According to the preset industry classification standard, obtain the report data sample of the environmental assessment data table in the scanned copy of the environmental assessment report of the specified industry, and use the report data sample to construct the image data set of the environmental assessment data table of the specified industry; Performing data preprocessing on the image data set of the environmental evaluation data table of the specified industry, and dividing the image data set of the environmental evaluation data table after the data preprocessing into a first training set and a first validation set; Establishing a table recognition neural network model based on the environmental assessment report, training the table recognition neural network model using the first training set, verifying the table recognition neural network model using the first verification set, and generating the table recognition neural network model that has been trained and verified; Using the trained table recognition neural network model to recognize the scanned copy of the environmental assessment report, and obtaining the table location and table row and column location in the environmental assessment report; According to the table positioning and table row and column positioning information in the environmental assessment report, image segmentation is performed on the table cells in the environmental assessment report to obtain a segmented table cell image in the environmental assessment report; Using the segmented table cell images in the environmental assessment report, a picture data set of table cell images in the environmental assessment report is established, and the picture data set of table cell images is divided into a second training set and a second validation set; Establishing a character recognition neural network model based on the environmental assessment report, training the character recognition neural network model using the second training set, verifying the character recognition neural network model using the second verification set, and generating the trained and verified character recognition neural network model; Using the trained and verified character recognition neural network model, perform text recognition on each cell in the environmental assessment report to obtain a text recognition result for each cell, and perform data correction and cleaning on the text recognition result for each cell; By using the table positioning and table row and column positioning information in the environmental assessment report and the text recognition results of each cell, the recognition content of the text recognition results is output and stored in a structured manner.

2. The method for character recognition and extraction of an environmental assessment report form according to claim 1, characterized in that: Data preprocessing of the image data set of the environmental assessment data table of the specified industry includes grayscale normalization; The formula for grayscale normalization is: ； In the formula, is the gray value of a single image pixel after normalization, is the gray value of a single image pixel to be processed in the original image, is the minimum gray value of the image to be processed, is the maximum grayscale value of the image to be processed, is the maximum value of the image interval to be processed, The minimum value of the image interval to be processed.

3. The method for character recognition and extraction of an environmental assessment report form according to claim 1, characterized in that: The data preprocessing of the image data set of the environmental assessment data table of the specified industry also includes data enhancement processing; the data enhancement processing includes performing at least one of random scaling, center cropping, random horizontal flipping, image denoising, binarization, and edge detection operations on the images in the image data set of the environmental assessment data table of the specified industry.

4. The method for character recognition and extraction of an environmental assessment report form according to claim 1, characterized in that: In the process of establishing a table recognition neural network model based on the environmental assessment report, the table recognition neural network model adopts a hybrid model of a fast convolutional neural network of a deformable CNN and a feature pyramid network. The expression formula of the hybrid model of a fast convolutional neural network of a deformable CNN and a feature pyramid network is: ； In the formula, * is the convolution operation, F is the filter, I For images, K Defined as 1 / 2 of the filter size, H is the image height, W is the image width, i , j To define where to perform the convolution, Defined as the vertical offset, Defined as a horizontal offset.

5. The method for character recognition and extraction of an environmental assessment report form according to claim 1, characterized in that: In the process of establishing a character recognition neural network model based on an environmental assessment report, the character recognition neural network model uses a deep residual network model as a feature extractor, and batch normalization is performed on the character recognition neural network model. The formula for batch normalization of the character recognition neural network model is: ； In the formula, is the output value of batch normalization, is the input value of batch normalization, is the mean of all samples in a specified batch, is the variance of all samples in the specified batch, and ϵ is a constant set to avoid division by zero; After batch normalization of the character recognition neural network model, the method further includes performing linear transformation and translation on the batch normalization results: ； In the formula, is the output value after linear transformation and translation, and γ and β are both learnable parameters.

6. The method for character recognition and extraction of environmental assessment report form according to claim 5, characterized in that: The process of establishing a character recognition neural network model based on an environmental assessment report also includes performing Dropout regularization on the character recognition neural network model: y=mask * x ; Where y is the model output parameter value; x Input parameter values ​​for the model. Mask is a vector of Boolean values ​​0 and 1, indicating whether each neuron is "pruned"; mask = Bernoulli(p,n); Where, Bernoulli is the discrete probability distribution function of a binary random variable, p is the probability, n is the number of neuron layers, and Bernoulli(p,n) means sampling a vector of size n with probability p.

7. The method for character recognition and extraction of environmental assessment report form according to claim 1, characterized in that: In the process of obtaining the table positioning and table row and column positioning in the environmental assessment report: Get the relative coordinate information of cells in the table, organize the relative coordinate information of cells and output it as structured data. The expression of structured data is [x0, y0, x1, y1, position]; The position of the cell is analyzed by a recursive algorithm to form position positioning information. Position is expressed as additional positioning information, which includes the row and column information of the cell and the parent cell information of the cell. The expression of the additional positioning information is [ID, H`, V, Parent, Type], where ID is the unique identifier of the cell; H` and V are the internal row and column information of the cell; Parent identifies the ID of the parent cell to which the cell belongs. If there is nested cell, use Parent to specify the ID of the parent cell to which the cell belongs. A Type flag of 1 indicates that the cell is a metadata cell, a Type flag of 0 indicates that the cell has an embedded cell, a Type flag of 2 indicates that the cell spans rows and columns, and a Type flag of -1 indicates that it is header data.

8. A text recognition and extraction device for an environmental assessment report form, characterized in that: include: A table image data set construction module is used to obtain report data samples of environmental assessment data tables in scanned copies of environmental assessment reports for a specified industry according to preset industry classification standards, and use the report data samples to construct an image data set of environmental assessment data tables for the specified industry; A table image data set processing module is used to perform data preprocessing on the image data set of the environmental evaluation data table of the specified industry, and divide the image data set of the environmental evaluation data table after data preprocessing into a first training set and a first verification set; A table recognition neural network model building module is used to establish a table recognition neural network model based on the environmental assessment report, train the table recognition neural network model using the first training set, verify the table recognition neural network model using the first verification set, and generate the table recognition neural network model that has been trained and verified; A positioning information acquisition module is used to use the trained table recognition neural network model to recognize the scanned copy of the environmental assessment report and obtain the table location and table row and column location in the environmental assessment report; The cell image processing module is used to perform image segmentation on the table cells in the environmental assessment report according to the table positioning and table row and column positioning information in the environmental assessment report, and obtain the segmented table cell images in the environmental assessment report; A cell image data set construction module is used to use the segmented table cell images in the environmental assessment report to establish an image data set of table cell images in the environmental assessment report, and divide the image data set of table cell images into a second training set and a second validation set; A character recognition neural network model building module, used to establish a character recognition neural network model based on the environmental assessment report, train the character recognition neural network model using the second training set, verify the character recognition neural network model using the second verification set, and generate the trained and verified character recognition neural network model; A cell text recognition processing module is used to use the trained and verified character recognition neural network model to perform text recognition on each cell in the environmental assessment report, obtain the text recognition result of each cell, and perform data correction and cleaning on the text recognition result of each cell; The cell text recognition result output module is used to utilize the table positioning and table row and column positioning information in the environmental assessment report, as well as the text recognition result of each cell, to output and store the recognition content of the text recognition result in a structured manner.

9. The device for recognizing and extracting characters from an environmental assessment report form according to claim 8, characterized in that: In the table image data set processing module, data preprocessing of the image data set of the environmental assessment data table of the specified industry includes grayscale normalization processing; The formula for grayscale normalization is: ； In the formula, is the gray value of a single image pixel after normalization, is the gray value of a single image pixel to be processed in the original image, is the minimum gray value of the image to be processed, is the maximum grayscale value of the image to be processed, is the maximum value of the image interval to be processed, is the minimum value of the image interval to be processed; In the table image data set processing module, the data preprocessing of the image data set of the environmental assessment data table of the specified industry also includes data enhancement processing; The data enhancement processing includes performing at least one of random scaling, center cropping, random horizontal flipping, image denoising, binarization, and edge detection operations on the images in the image data set of the environmental evaluation data table of the specified industry.

10. The device for recognizing and extracting characters from an environmental assessment report form according to claim 8, characterized in that: In the table recognition neural network model building module: The table recognition neural network model adopts a hybrid model of a fast convolutional neural network of a deformable CNN and a feature pyramid network. The expression formula of the hybrid model of a fast convolutional neural network of a deformable CNN and a feature pyramid network is: ； In the formula, * is the convolution operation, F is the filter, I For images, K Defined as 1 / 2 of the filter size, H is the image height, W is the image width, i , j To define where to perform the convolution, Defined as the vertical offset, Defined as horizontal offset; In the character recognition neural network model building module: The character recognition neural network model uses a deep residual network model as a feature extractor, and performs batch normalization on the character recognition neural network model. The formula for performing batch normalization on the character recognition neural network model is: ； In the formula, is the output value of batch normalization, is the input value of batch normalization, is the mean of all samples in a specified batch, is the variance of all samples in the specified batch, and ϵ is a constant set to avoid division by zero; In the character recognition neural network model building module, the batch normalized results are linearly transformed and translated: ； In the formula, is the output value after linear transformation and translation, γ and β are both learnable parameters; The character recognition neural network model construction module also includes performing Dropout regularization on the character recognition neural network model: y=mask * x ; Where y is the model output parameter value; x Input parameter values ​​for the model. Mask is a vector of Boolean values ​​0 and 1, indicating whether each neuron is "pruned"; mask = Bernoulli(p,n); Where, Bernoulli is the discrete probability distribution function of a binary random variable, p is the probability, n is the number of neuron layers, and Bernoulli(p,n) means sampling a vector of size n with probability p; In the cell text recognition processing module: Get the relative coordinate information of cells in the table, organize the relative coordinate information of cells and output it as structured data. The expression of structured data is [x0, y0, x1, y1, position]; The position of the cell is analyzed by a recursive algorithm to form position positioning information. Position is expressed as additional positioning information, which includes the row and column information of the cell and the parent cell information of the cell. The expression of the additional positioning information is [ID, H`, V, Parent, Type], where ID is the unique identifier of the cell; H` and V are the internal row and column information of the cell; Parent identifies the ID of the parent cell to which the cell belongs. If there is nested cell, use Parent to specify the ID of the parent cell to which the cell belongs. A Type flag of 1 indicates that the cell is a metadata cell, a Type flag of 0 indicates that the cell has an embedded cell, a Type flag of 2 indicates that the cell spans rows and columns, and a Type flag of -1 indicates that it is header data.

Images