Document outline extraction model construction method, device, equipment and readable storage medium

Abstract

The present application relates to a method, apparatus, device and readable storage medium for constructing a document contour extraction model, and relates to the field of document detection technology, including obtaining an image training set, the image training set including multiple sample images; labeling each sample image to obtain a corresponding mask label, and determining the vertex information of the corresponding sample image based on the mask label to form a new image training set; training a semantic segmentation model based on the new image training set to generate a document contour extraction model, the semantic segmentation model including a learning downsampling module, a global feature extraction module, a feature fusion module and a standard classifier, the standard classifier including a vertex regression submodule. The document contour extraction model generated by the present application implements a vertex prediction function while segmenting the document area in the image, and can then extract the document contour by directly obtaining the vertex information of the document area, effectively improving the accuracy of document contour information extraction.

Description

Technical Field

[0001] The present application relates to the field of document detection technology, and in particular to a method, device, equipment and readable storage medium for constructing a document contour extraction model. Background Art

[0002] With the improvement of mobile phone performance and the rapid development of artificial intelligence, document recognition and subsequent processing technologies, including handwriting location, error erasure, optical character recognition (OCR), sentiment analysis, and word segmentation in natural language processing (NLP), are gradually being applied to students' daily learning. However, directly performing handwriting location, error erasure, OCR, sentiment analysis, and word segmentation on photographed documents can yield unsatisfactory results due to the susceptibility of these processing methods to factors such as the photographed background and environment. Therefore, it is crucial to detect regions in photographed documents, obtain their contour information, and then perform corrections on them. The main goal of document contour extraction is to obtain the document's contour information through an algorithm, then remove background information from the photograph based on this contour information, and finally use this contour information to correct the document region in the photograph, thereby improving the accuracy of subsequent processing of the photographed document.

[0003] In related technologies, methods for document contour extraction can be divided into: 1. Segment the document area through semantic segmentation methods, and then use traditional image processing methods to analyze the results of semantic segmentation and calculate the contour information; 2. First obtain the text area information in the image through text area detection, and then calculate the offset based on the center point and the text area to obtain the contour information.

[0004] However, because semantic segmentation can produce irregular shapes, traditional image processing methods may not accurately capture contour information. Furthermore, photographed text also exhibits irregularities, so calculating the contour using only the offset between the center point and the text area detected will result in poor accuracy. Therefore, current methods for document contour extraction suffer from inaccuracy. Summary of the Invention

[0005] The present application provides a document contour extraction model construction method, apparatus, device and readable storage medium to solve the problem of poor accuracy in document contour information extraction existing in the related art.

[0006] In a first aspect, a method for constructing a document outline extraction model is provided, comprising the following steps:

[0007] Obtaining a picture training set, wherein the picture training set includes multiple sample pictures;

[0008] Label each sample image to obtain a corresponding mask label, and determine the vertex information of the corresponding sample image based on the mask label to form a new image training set;

[0009] The semantic segmentation model is trained based on the new image training set to generate a document outline extraction model. The semantic segmentation model includes a learning downsampling module, a global feature extraction module, a feature fusion module and a standard classifier. The standard classifier includes a vertex regression submodule.

[0010] In some embodiments, the loss function L in the standard classifier is calculated as follows:

[0011] L=ω1·L point +ω2·L mask

[0012] Where ω1 represents the vertex weight, L point represents the loss between the predicted vertex enclosed area and the true vertex enclosed area, ω2 represents the mask weight, L mask Represents the loss of the mask.

[0013] In some embodiments, after the step of determining the vertex information of the corresponding sample image according to the mask label to form a new image training set, the method further includes:

[0014] Determining whether a sample image in the new image training set contains the entire document area;

[0015] If so, performing a first enhancement process on the sample image in the new image training set to obtain a first sample image;

[0016] If not, a second enhancement process is performed on the sample image in the new image training set to obtain a second sample image.

[0017] In some embodiments, the semantic segmentation model further includes a splicing module, which is used to splice the high-level semantic information output by the feature fusion module and the low-level semantic information output by the learning downsampling module to obtain spliced ​​information, and the spliced ​​information serves as the input of the standard classifier.

[0018] In some embodiments, labeling each sample image to obtain a corresponding mask label includes:

[0019] Mark the vertices of the document area of ​​each sample image, and create a mask background image for each sample image based on the original image information;

[0020] The area enclosed by multiple vertices on the corresponding sample image is filled based on the mask background image to obtain a mask label.

[0021] In some embodiments, after the step of training the semantic segmentation model based on the new image training set to generate the document outline extraction model, the method further includes:

[0022] Obtaining a picture to be predicted and performing normalization processing on the picture to be predicted;

[0023] The normalized image to be predicted is input into the document contour extraction model to obtain document contour information of the image to be predicted and information of multiple vertices on the document contour.

[0024] In some embodiments, after the step of inputting the normalized image to be predicted into the document outline extraction model to obtain document outline information of the image to be predicted and information of multiple vertices on the document outline, the method further includes:

[0025] Calculate the minimum circumscribed rectangle of the area enclosed by the multiple vertices on the document outline according to the multiple vertex information on the document outline;

[0026] Calculating a perspective transformation matrix based on vertex information of the minimum circumscribed rectangle and multiple vertex information on the document outline;

[0027] The image to be predicted is corrected according to the perspective transformation matrix, and a document area of ​​the image to be predicted is output.

[0028] In a second aspect, a document outline extraction model construction device is provided, comprising:

[0029] an acquisition unit, configured to acquire a picture training set, wherein the picture training set includes a plurality of sample pictures;

[0030] A processing unit, which is used to label each sample image to obtain a corresponding mask label, and determine the vertex information of the corresponding sample image according to the mask label to form a new image training set;

[0031] A training unit is used to train a semantic segmentation model based on the new image training set to generate a document outline extraction model, wherein the semantic segmentation model includes a learning downsampling module, a global feature extraction module, a feature fusion module and a standard classifier, and the standard classifier includes a vertex regression submodule.

[0032] In a third aspect, a document contour extraction model construction device is provided, comprising: a memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the aforementioned document contour extraction model construction method.

[0033] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the aforementioned document contour extraction model construction method is implemented.

[0034] The beneficial effects brought about by the technical solution provided by this application include: effectively improving the accuracy of document contour information extraction.

[0035] The present application provides a method, apparatus, device and readable storage medium for constructing a document contour extraction model, including obtaining a picture training set, wherein the picture training set includes multiple sample pictures; annotating each sample picture to obtain a corresponding mask label, and determining each vertex information of the corresponding sample picture based on the mask label to form a new picture training set; training a semantic segmentation model based on the new picture training set to generate a document contour extraction model, wherein the semantic segmentation model includes a learning downsampling module, a global feature extraction module, a feature fusion module and a standard classifier, and the standard classifier includes a vertex regression submodule. Since the present application integrates a vertex regression submodule in the standard classifier, the generated document contour extraction model realizes the vertex prediction function while segmenting the document area in the picture, and can then extract the document contour by directly obtaining the vertex information of the document area, which is not affected by whether the graphics are regular or whether the text material is regular, and effectively improves the accuracy of document contour information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0037] Figure 1 A flowchart of a document outline extraction model construction method provided in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of the structure of the document outline extraction model provided in the embodiment of the present application;

[0039] Figure 3 A schematic diagram of the process of document contour extraction and image correction provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of the structure of a document outline extraction model construction device provided in an embodiment of the present application;

[0041] Figure 5A structural diagram of a document outline extraction model building device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] The embodiments of the present application provide a document contour extraction model construction method, apparatus, device and readable storage medium, which can solve the problem of poor accuracy in document contour information extraction existing in related technologies.

[0044] Figure 1 A document outline extraction model construction method provided in an embodiment of the present application includes the following steps:

[0045] Step S10: obtaining a picture training set, wherein the picture training set includes a plurality of sample pictures;

[0046] Step S20: labeling each sample image to obtain a corresponding mask label, and determining each vertex information of the corresponding sample image according to the mask label to form a new image training set;

[0047] Furthermore, in the embodiment of the present application, the step of labeling each sample image to obtain a corresponding mask label includes:

[0048] Mark the vertices of the document area of ​​each sample image, and create a mask background image for each sample image based on the original image information;

[0049] The area enclosed by multiple vertices on the corresponding sample image is filled based on the mask background image to obtain a mask label.

[0050] Exemplarily, in this embodiment, each sample image in the image training set needs to be preprocessed first to generate the training samples required for model training; specifically, a data annotation tool (such as Labelme, which is an offline image annotation tool) can be used to perform multi-vertex annotation on the document area in each sample image, and create a Mask background image based on the original image information, and fill the area enclosed by multiple vertices with the Mask background image to generate a Mask label, and then traverse all vertices, calculate the four vertices with the largest area, and extract the annotation type, and finally generate the samples and labels required for model training.

[0051] Furthermore, in the embodiment of the present application, after step S20, the following steps are further included:

[0052] Determining whether a sample image in the new image training set contains the entire document area;

[0053] If so, performing a first enhancement process on the sample image in the new image training set to obtain a first sample image;

[0054] If not, a second enhancement process is performed on the sample image in the new image training set to obtain a second sample image.

[0055] For example, in the related art of document contour extraction methods using traditional image processing, the image must first be binarized, and then the binarization result is used to perform edge detection on the image, mainly to find areas where pixel values ​​change dramatically. Then, based on the edge information in the image, closed edge areas in the image are found as the found contour areas, and finally, the vertex coordinates of the contour are output to complete the contour extraction. The inventors found that when using the above-mentioned traditional image processing methods to process images and calculate contour information, the document materials obtained are quite different due to the wide range of hardware of the camera equipment, the time of shooting, and the scene in which the camera is taken. Therefore, if only traditional image processing methods are used to obtain document contour information, it cannot be applied to all types of document materials.

[0056] Therefore, after analyzing various scenarios, this embodiment performs document differentiation processing on the above-mentioned training sample data according to the application scenario, that is, distinguishing between pictures with complete document outlines and partial document pictures with incomplete outlines, so that this method is applicable to the outline extraction of various document materials obtained by photographing. Specifically, before model training, data enhancement processing is performed on the picture samples whether or not they contain the entire document area. The picture samples containing the entire document area are subjected to the first enhancement processing, that is, rotation, inversion, color transformation and normalization and other enhancement processing, while the picture samples that do not contain the entire document area are subjected to the second enhancement processing, that is, translation, rotation, inversion, color transformation and normalization and other enhancement processing. After the model is trained with the enhanced picture samples, the obtained model can be applicable to all kinds of scenarios, which greatly increases the robustness of the model. It can be seen that the embodiment of the present application performs separate processing on picture data that does not include complete boundary information, and enables the model to process various types of picture data through different data enhancement methods without reducing the accuracy of picture data containing complete boundary information.

[0057] Step S30: training a semantic segmentation model based on the new image training set to generate a document outline extraction model, wherein the semantic segmentation model includes a learning downsampling module, a global feature extraction module, a feature fusion module and a standard classifier, and the standard classifier includes a vertex regression submodule.

[0058] For example, Fast SCNN (Fast Semantic Segmentation Network) is a deep convolutional neural network model for semantic segmentation tasks. Its input is a normalized three-channel image tensor, and its output is a binary mask with the same length and width as the input. The value of each position represents whether the corresponding pixel at that position is part of the document area. It has the characteristics of fast speed, low computational overhead and high accuracy. At the same time, the Fast SCNN model uses features from different stages for learning. If needed, it can also add lower-level features than those used in the existing Fast SCNN network structure. It can better handle the different features of the contour and content in the document material. The semantic segmentation model designed in this embodiment needs to output document area and vertex information, among which the region extraction part is used to segment the document area as a whole in the input image. Therefore, this embodiment improves the Fast SCNN model to form a new semantic segmentation model, and trains the new semantic segmentation model through the aforementioned training samples to generate a document contour extraction model.

[0059] Specifically, the traditional Fast SCNN network structure only includes a learning downsampling module, a global feature extraction module, a feature fusion module and a standard classifier, while Figure 2 As shown, this embodiment not only has the above-mentioned modules, but also adds a vertex regression output branch to the Classifier (i.e., the standard classifier) ​​in order to better segment the document area and obtain the vertex information of the document: the vertex regression sub-module, that is, the vertex information of the document area in the input image is predicted by the vertex regression sub-module. It can be seen that since the present application integrates the vertex regression sub-module in the standard classifier, the generated document contour extraction model realizes the vertex prediction function while segmenting the document area in the image, and then can extract the document contour by directly obtaining the vertex information of the document area. It is not affected by whether the graphics are regular or whether the text material is regular, which effectively improves the accuracy of document contour information extraction.

[0060] Furthermore, in an embodiment of the present application, the semantic segmentation model also includes a splicing module, which is used to splice the high-level semantic information output by the feature fusion module and the low-level semantic information output by the learning downsampling module to obtain splicing information, and the splicing information is used as the input of the standard classifier.

[0061] For example, in order to more accurately obtain the boundary information of the document to extract the contour area, after analyzing the characteristics of the sample data that the model needs to process, it is found that in order to better extract the contour information model, it is necessary not only to learn the high-level feature (i.e., high-level semantic information) within the document area, but also to learn the low-level feature (i.e., low-level semantic information) of the document boundary. This is because if only the high-level feature within the document area is learned, it will lead to inaccurate contour extraction when the color of the document area is similar to the background color. Therefore, in this embodiment, after the Feature Fusion (i.e., feature fusion module) of Fast SCNN outputs the high-level feature, the high-level feature is spliced ​​with the low-level feature output by the learning downsampling module through the splicing module (i.e., Figure 2 The concatenated features are then sent to the Classifier for processing, thereby improving the accuracy of document area contour extraction. It can be seen that the embodiment of the present application ingeniously proposes a FastSCNN network structure that increases low-level feature learning based on the features of the photographed image containing the document area.

[0062] Furthermore, in the embodiment of the present application, the calculation formula of the loss function L in the standard classifier is as follows:

[0063] L=ω1·L point +ω2·L mask

[0064] Where ω1 represents the vertex weight, L point represents the loss between the predicted vertex enclosed area and the true vertex enclosed area, ω2 represents the mask weight, L mask Represents the loss of the mask.

[0065] For example, this embodiment also modifies the calculation formula of the loss function L in the standard classifier to:

[0066] L=ω1·L point +ω2·L mask

[0067] Where ω1 represents the vertex weight, L point Can be L GIoU , which represents the GIoU loss between the predicted vertex enclosed area and the true vertex enclosed area (GIoU loss is a generalized IoU (Intersection over Union) loss), ω2 represents the mask weight, L mask Can be LCrossEntropy , which represents the cross entropy loss of different class masks.

[0068] Among them, L CrossEntropy The specific calculation formula is as follows:

[0069]

[0070] In the formula, loss represents the cross entropy loss function, x represents the document area, and class represents the Mask type.

[0071] L GIoU The specific calculation formula is as follows:

[0072] L GIoU =1-(GIoU)

[0073]

[0074] Where IoU represents the intersection-over-union loss, which can reflect the detection effect of the predicted detection box and the real detection box. c It represents the area of ​​the rectangle that encloses the union area of ​​the real vertex and the predicted vertex with the minimum rectangle, and u represents the area of ​​the union area of ​​the real vertex and the predicted vertex.

[0075] After the above training, the embodiment of the present application can finally obtain the optimal document contour extraction model with an accuracy rate of 98.59%. Through this model, the document area contour and vertex information in the image can be accurately output, and in the process of model training, the L in the loss function is point and L mask It plays a mutually reinforcing role and improves the prediction effect of the model, that is, the two different outputs (i.e., document area contour and vertex information) are also in a mutually reinforcing process.

[0076] Furthermore, in the embodiment of the present application, after step S30, the following steps are further included:

[0077] Obtaining a picture to be predicted and performing normalization processing on the picture to be predicted;

[0078] The normalized image to be predicted is input into the document contour extraction model to obtain document contour information of the image to be predicted and information of multiple vertices on the document contour.

[0079] For demonstration, this embodiment first normalizes the image to be predicted in order to convert it into the image format expected by the document contour extraction model; then the normalized image to be predicted is input into the document contour extraction model to output the prediction result, that is, the document contour information of the image to be predicted and the multiple vertex information on the document contour.

[0080] Furthermore, in an embodiment of the present application, after the step of inputting the normalized image to be predicted into the document contour extraction model to obtain document contour information of the image to be predicted and information of multiple vertices on the document contour, the method further includes:

[0081] Calculate the minimum circumscribed rectangle of the area enclosed by the multiple vertices on the document outline according to the multiple vertex information on the document outline;

[0082] Calculating a perspective transformation matrix based on vertex information of the minimum circumscribed rectangle and multiple vertex information on the document outline;

[0083] The image to be predicted is corrected according to the perspective transformation matrix, and a document area of ​​the image to be predicted is output.

[0084] Exemplarily, the image to be predicted is corrected according to the vertex information on the acquired document contour to extract the document contour area. Specifically, the coordinates of each vertex on the document contour are first verified. If the vertex coordinates are valid (that is, all coordinates are within the corresponding document area), the size of the minimum circumscribed rectangle of the area enclosed by the vertex coordinates is calculated based on the vertex coordinates, and the perspective transformation matrix is ​​calculated based on the vertex coordinates of the minimum circumscribed rectangle and the vertex coordinates on the document contour. The perspective transformation matrix is ​​then used to correct the image to be predicted, and the document area of ​​the image to be predicted can be output.

[0085] The following combination Figure 3 Several parts of the document contour extraction model of the embodiment of this application are explained, including data preprocessing, network structure adjustment and model training, document contour and vertex prediction, and contour extraction and image correction.

[0086] Step N1: Input multiple sample images taken in various scenes containing document areas.

[0087] Step N2: Use a labeling tool to label each sample image, obtain a mask label for the sample image, and generate four vertices that can contain the mask label area based on the labeling results. The mask label is the area in the sample image with complete boundary information. If complete boundary information does not exist, the mask label is the entire document area in the sample image. The four vertices that contain the mask label area are the four vertices that can enclose the largest area after traversing and calculating all vertices in the sample image.

[0088] Step N3: Determine whether the sample image contains the entire document region. Perform enhancement processing such as rotation, inversion, color conversion, and normalization on the sample image containing the entire document region. Perform enhancement processing such as translation, rotation, inversion, color conversion, and normalization on the sample image not containing the entire document region. The method for determining whether the sample image contains the entire document region is to use a unified naming method to distinguish sample images containing the entire document region during the sample annotation process. That is, whether the sample image contains the entire document region can be determined simply by the name of the sample image.

[0089] Step N4: All the training samples obtained in step N3 are input into the semantic segmentation network structure with the output branches of low-level features and vertex regression added and the calculation method of the loss function modified for model training. Finally, the optimal document outline extraction model with an accuracy of 98.59% is obtained.

[0090] Step N5: Obtain the image to be predicted and perform corresponding processing on it. The processing performed on the image to be predicted is the normalization processing performed on the validation set input image during the model training process, in order to convert the image to be predicted into the image format expected by the model prediction.

[0091] Step N6: Use the document contour extraction model obtained in step N4 to predict the image to be predicted input in step N5, and output the prediction result, namely the document contour information and multiple vertex information on the document contour.

[0092] Step N7: Correct the image to be predicted using the vertex information obtained in step N6 to extract the document contour area of ​​the image to be predicted.

[0093] It can be seen that the embodiment of the present application provides a method for directly outputting contour information by predicting the four vertices of the document contour. Compared with the traditional image processing method, the method can be applied to document materials obtained from various types of photographic scenes, and is also applicable to document materials that do not contain the entire document area and cannot be processed by traditional image processing; and for the method of segmenting the document area through the semantic segmentation method and then using traditional image processing to analyze the results of the semantic segmentation and calculate the contour information, the result obtained by semantic segmentation may be an irregular graphic, and accurate results cannot be obtained using traditional image processing analysis. The embodiment of the present application can directly obtain the vertex information of the document area to extract the document contour, thereby avoiding the occurrence of such problems; in addition, for the problem of low accuracy caused by first obtaining the text area information in the picture through text area detection and calculating the offset according to the center point and the text area to obtain the contour information, the embodiment of the present application can directly segment the document area in the picture and add a vertex prediction function on this basis, thereby ignoring the adverse effects brought about by various irregularities of the text material.

[0094] See also Figure 4 As shown, the embodiment of the present application also provides a document outline extraction model construction device, including:

[0095] an acquisition unit, configured to acquire a picture training set, wherein the picture training set includes a plurality of sample pictures;

[0096] A processing unit, which is used to label each sample image to obtain a corresponding mask label, and determine the vertex information of the corresponding sample image according to the mask label to form a new image training set;

[0097] A training unit is used to train a semantic segmentation model based on the new image training set to generate a document outline extraction model, wherein the semantic segmentation model includes a learning downsampling module, a global feature extraction module, a feature fusion module and a standard classifier, and the standard classifier includes a vertex regression submodule.

[0098] Since this application integrates a vertex regression sub-module into the standard classifier, the generated document contour extraction model realizes the vertex prediction function while segmenting the document area in the image, and can extract the document contour by directly obtaining the vertex information of the document area. It is not affected by whether the graphics are regular or the text material is regular, which effectively improves the accuracy of document contour information extraction.

[0099] Furthermore, in the embodiment of the present application, the calculation formula of the loss function L in the standard classifier is as follows:

[0100] L=ω1·L point +ω2·L mask

[0101] Where ω1 represents the vertex weight, L point represents the loss between the predicted vertex enclosed area and the true vertex enclosed area, ω2 represents the mask weight, L mask Represents the loss of the mask.

[0102] Furthermore, in the embodiment of the present application, the processing unit is further configured to:

[0103] Determining whether a sample image in the new image training set contains the entire document area;

[0104] If so, performing a first enhancement process on the sample image in the new image training set to obtain a first sample image;

[0105] If not, a second enhancement process is performed on the sample image in the new image training set to obtain a second sample image.

[0106] Furthermore, in an embodiment of the present application, the semantic segmentation model also includes a splicing module, which is used to splice the high-level semantic information output by the feature fusion module and the low-level semantic information output by the learning downsampling module to obtain splicing information, and the splicing information is used as the input of the standard classifier.

[0107] Furthermore, in the embodiment of the present application, the processing unit is specifically configured to:

[0108] Mark the vertices of the document area of ​​each sample image, and create a mask background image for each sample image based on the original image information;

[0109] The area enclosed by multiple vertices on the corresponding sample image is filled based on the mask background image to obtain a mask label.

[0110] Furthermore, in the embodiment of the present application, the processing unit is further configured to:

[0111] Obtaining a picture to be predicted and performing normalization processing on the picture to be predicted;

[0112] The normalized image to be predicted is input into the document contour extraction model to obtain document contour information of the image to be predicted and information of multiple vertices on the document contour.

[0113] Furthermore, in an embodiment of the present application, the apparatus further includes a correction unit, which is configured to:

[0114] Calculate the minimum circumscribed rectangle of the area enclosed by the multiple vertices on the document outline according to the multiple vertex information on the document outline;

[0115] Calculating a perspective transformation matrix based on vertex information of the minimum circumscribed rectangle and multiple vertex information on the document outline;

[0116] The image to be predicted is corrected according to the perspective transformation matrix, and a document area of ​​the image to be predicted is output.

[0117] For example, see Figure 2 As shown, the document outline extraction model formed in the embodiment of the present application includes a learning down-sampling module (Learning to Down-sample), a global feature extraction module (Global Feature Extractor), a feature fusion module (Feature Fusion), a splicing module and a standard classifier (Classifier). Figure 2Different shapes represent different objects, for example, a triangle represents a ConV2D convolution layer, and a cuboid represents a DSConV depth-separable convolution layer. Specifically, the learning downsampling module includes an input layer (i.e., Input), a common ConV2D convolution layer, and two DSConV depth-separable convolution layers. The global feature extraction module includes eight Bottleneck convolution layers and one PyramidPooling convolution layer, and the PyramidPooling convolution layer is used to extract contextual features of different scales. The feature fusion module includes one Upsample upsampling convolution layer, one DWConv convolution layer, and two Co nv2D convolution layer; the splicing module is used to splice the high-level semantic information output by the feature fusion module and the low-level semantic information output by the learning downsampling module to obtain splicing information, which is used as the input of the standard classifier; the standard classifier includes 2 DSConV depth-separable convolution layers, 1 ConV2D convolution layer, 1 Upsample upsampling convolution layer, 1 vertex regression submodule and output layer (i.e., Softmax, Softmax is a normalized exponential function). The vertex regression submodule includes 1 DSConV depth-separable convolution layer, which is used to regress vertices and output point information to predict the vertex information of the document in the image to be predicted.

[0118] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned document contour extraction model construction method embodiment, and will not be repeated here.

[0119] The apparatus provided in the above embodiment can be implemented in the form of a computer program. The computer program can be used in Figure 5 The document outline extraction model shown is built and run on the device.

[0120] An embodiment of the present application also provides a document contour extraction model construction device, comprising: a memory, a processor, and a network interface connected via a system bus, wherein at least one instruction is stored in the memory, and at least one instruction is loaded and executed by the processor to implement all or part of the steps of the aforementioned document contour extraction model construction method.

[0121] Among them, the network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0122] The processor may be a CPU, or other general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor, or any conventional processor. The processor is the control center of a computer device, connecting various parts of the entire computer device using various interfaces and lines.

[0123] The memory can be used to store computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function (such as a video playback function, an image playback function, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as video data, image data, etc.). In addition, the memory can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0124] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, all or part of the steps of the aforementioned document contour extraction model construction method are implemented.

[0125] The embodiments of the present application implement all or part of the aforementioned processes, and may also be completed by instructing related hardware through a computer program. The computer program may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of each of the above methods may be implemented. Among them, the computer program includes computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0126] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, servers, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage) containing computer-usable program code.

[0127] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0129] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0130] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A document outline extraction model construction method, characterized in that: The following steps are involved: Obtaining a picture training set, wherein the picture training set includes multiple sample pictures; Label each sample image to obtain a corresponding mask label, and determine the vertex information of the corresponding sample image based on the mask label to form a new image training set; Training a semantic segmentation model based on the new image training set to generate a document outline extraction model, wherein the semantic segmentation model includes a learning downsampling module, a global feature extraction module, a feature fusion module, and a standard classifier, wherein the standard classifier includes a vertex regression submodule; The calculation formula of the loss function L in the standard classifier is as follows: L=ω1·L point +ω2·L mask Where ω1 represents the vertex weight, L point represents the loss between the predicted vertex enclosed area and the true vertex enclosed area, ω2 represents the mask weight, L mask Indicates the loss of the mask; After the step of training the semantic segmentation model based on the new image training set to generate a document outline extraction model, the method further includes: Obtaining a picture to be predicted and performing normalization processing on the picture to be predicted; Inputting the normalized image to be predicted into the document contour extraction model to obtain document contour information of the image to be predicted and information of multiple vertices on the document contour; After the step of inputting the normalized image to be predicted into the document contour extraction model to obtain document contour information of the image to be predicted and information of multiple vertices on the document contour, the method further includes: Calculate the minimum circumscribed rectangle of the area enclosed by the multiple vertices on the document outline according to the multiple vertex information on the document outline; Calculating a perspective transformation matrix based on vertex information of the minimum circumscribed rectangle and multiple vertex information on the document outline; The image to be predicted is corrected according to the perspective transformation matrix, and a document area of ​​the image to be predicted is output.

2. The document outline extraction model construction method according to claim 1, characterized in that: After the step of determining the vertex information of the corresponding sample image according to the mask label to form a new image training set, the method further includes: Determining whether a sample image in the new image training set contains the entire document area; If so, performing a first enhancement process on the sample image in the new image training set to obtain a first sample image; If not, a second enhancement process is performed on the sample image in the new image training set to obtain a second sample image.

3. The document outline extraction model construction method according to claim 1, wherein: The semantic segmentation model also includes a splicing module, which is used to splice the high-level semantic information output by the feature fusion module and the low-level semantic information output by the learning downsampling module to obtain splicing information, and the splicing information is used as the input of the standard classifier.

4. The document outline extraction model construction method according to claim 1, wherein: The step of labeling each sample image to obtain a corresponding mask label includes: Mark the vertices of the document area of ​​each sample image, and create a mask background image for each sample image based on the original image information; The area enclosed by multiple vertices on the corresponding sample image is filled based on the mask background image to obtain a mask label.

5. A document outline extraction model construction device, characterized in that: include: an acquisition unit, configured to acquire a picture training set, wherein the picture training set includes a plurality of sample pictures; A processing unit, which is used to label each sample image to obtain a corresponding mask label, and determine the vertex information of the corresponding sample image according to the mask label to form a new image training set; A training unit, configured to train a semantic segmentation model based on the new image training set to generate a document outline extraction model, wherein the semantic segmentation model includes a learning downsampling module, a global feature extraction module, a feature fusion module, and a standard classifier, wherein the standard classifier includes a vertex regression submodule; The calculation formula of the loss function L in the standard classifier is as follows: L=ω1·L point +ω2·L mask Where ω1 represents the vertex weight, L point represents the loss between the predicted vertex enclosed area and the true vertex enclosed area, ω2 represents the mask weight, L mask Indicates the loss of the mask; The processing unit is further configured to: Obtaining a picture to be predicted and performing normalization processing on the picture to be predicted; Inputting the normalized image to be predicted into the document contour extraction model to obtain document contour information of the image to be predicted and information of multiple vertices on the document contour; The device further comprises a correction unit, which is configured to: Calculate the minimum circumscribed rectangle of the area enclosed by the multiple vertices on the document outline according to the multiple vertex information on the document outline; Calculating a perspective transformation matrix based on vertex information of the minimum circumscribed rectangle and multiple vertex information on the document outline; The image to be predicted is corrected according to the perspective transformation matrix, and a document area of ​​the image to be predicted is output.

6. A document outline extraction model construction device, characterized in that: include: A memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the document outline extraction model construction method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the document outline extraction model construction method according to any one of claims 1 to 4 is implemented.

Images