Method and model for extracting document from image

Abstract

The invention relates to the related field of image document processing, and discloses a method and a model for extracting a document from an image, target scene fitting is carried out by adopting a deep learning supervision mode, a data set source is a synthesis mode, the capability of generalization to a real target scene is realized, and the method and the model have high practicability. The randomness and the content complexity of a Chinese case in the image can be effectively handled, and the image document area needing to be identified is efficiently and quickly positioned and matched.

Description

Technical Field

[0001] The present invention relates to the field of image document processing, and in particular to a method and a model for extracting documents from images. Background Art

[0002] In the current era of rapid expansion of informatization, documents are an important way to transfer information between people and between people and devices. The way the content in documents is carried can be varied, depending on the source and purpose. The inherent file carrier format established by file management related software can be directly recognized by other devices during transmission, while some image carrier files captured by the device cannot be directly read when used, so a priority content recognition process is required.

[0003] When using the technology of extracting documents from images in the prior art, there is a low recognition rate and it is easy to cause misrecognition due to the randomness of the text in the image and the complexity of the content. Therefore, when facing complex recognition scenarios, how to set the recognition execution method is very important for optimizing the recognition effect. Summary of the invention

[0004] The purpose of the present invention is to provide a method and model for extracting documents from images to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for extracting a document from an image, comprising:

[0007] Based on the feature pyramid network FPN, image scaling is performed at different layers to obtain multi-scale feature information;

[0008] Further enhancing feature extraction of the feature information by a single-stage detector SSH, wherein the single-stage detector SSH does not include a fully connected layer and has an efficient processing speed;

[0009] It can jointly optimize the network layer through multiple tasks based on the multi-head self-attention mechanism Multi-Head to optimize overfitting and generalization performance.

[0010] As a further solution of the present invention: the step of performing multi-task joint optimization of the network layer based on the multi-head self-attention mechanism Multi-Head specifically includes a specific technical solution:

[0011] The DSNT of spatial transformation coordinates enables the model to directly predict the position of key points, so that it can be differentiated from the output end to the input end, and the weight learning can be updated by back propagation;

[0012] Perform image segmentation judgment. If the image needs to be segmented, the target area and the background in the image are distinguished and segmented through MASK, that is, the target area and non-target area are masked and classified;

[0013] The center point of the document is predicted based on the heatmap to obtain the center point coordinate information. The center point prediction is auxiliary information used to improve the accuracy of object detection.

[0014] As a further aspect of the present invention: the DSNT includes a loss function:

[0015] The loss function is expressed as:

[0016] in The output heatmap used to characterize the network SSH has a shape of Nx64x48x32.

[0017] The p is used to represent the coordinates of the four corner points of the document, expressed as: p(x|y);

[0018] The λ is used to characterize the weight, indicating L euc The impact of this model on the weights learned;

[0019] Said Used to characterize the coordinates of the four corner points predicted after dsnt processing, expressed as μ;

[0020] The L euc Used to represent the Euclidean loss function, used for regression coordinates, formula: L euc (μ|p)=||μ―p||2, directly calculate the two-dimensional Euclidean distance between μ and p;

[0021] The L reg Used to characterize the regularity constraint of the distribution, and to constrain the generated heat map to be closer to the Gaussian distribution. The formula is: Where D(·||·) represents the KL divergence measurement, which is widely used to measure the similarity between two distributions. represents a bivariate Gaussian distribution, p represents the coordinates of the corner point, Standard deviation is the hyperparameter for generating Gauss heatmap, in pixels.

[0022] As a further solution of the present invention: the MASK includes a loss function:

[0023] The loss function is:

[0024] Among them, the yi Used to represent the label of sample i, with positive class: 1, negative class: 0;

[0025] The p(y i ) is used to represent the probability of sample i being predicted as a positive class, which is the number after sigmoid;

[0026] The N is used to represent the number of samples, which here refers to the number of pixels in the image. After summing up, the average is taken.

[0027] As a further solution of the present invention: the document center prediction includes a loss function:

[0028] The loss function is: focal +λ1L size +λ2L off ,

[0029] Among them, the L focal :(1―p t ) γ log(p t ), p t Represents the predicted probability; γ is a hyperparameter whose control range is [0,5], which controls the contribution of easy-to-classify samples to the loss, thereby increasing the loss ratio of difficult-to-classify samples;

[0030] The L size :|p size ―gt size |, p size Indicates the height and width of the predicted document; gt size Indicates the height and width of the actual document; the height and width here are calculated based on the center point of the document;

[0031] Said Downsampling will produce float, which needs to be converted to Int. The difference between the two is Localoffset. Indicates the predicted local offset; p indicates the coordinates of the center point before scaling; R indicates the outputstride (output stride); Indicates the center point corresponding to downsampling, type is Int.

[0032] The embodiment of the present invention aims to provide a method and model for extracting documents from images, comprising:

[0033] The key point prediction unit is used to enable the model to directly predict the key point position through the DSNT of the spatial transformation coordinates, so that it can be differentiated from the output end to the input end, and the weight learning can be updated by back propagation;

[0034] A segmentation judgment unit is used to perform image segmentation judgment. If the image needs to be segmented, the target area and the background in the image are distinguished and segmented by MASK, that is, the target area and the non-target area are masked and classified;

[0035] The center point prediction unit is used to predict the center point of the document based on the heatmap to obtain the center point coordinate information. The center point prediction is auxiliary information used to improve the object detection accuracy.

[0036] As a further solution of the present invention: the optimization fitting module includes:

[0037] The key point prediction unit is used to enable the model to directly predict the key point position through the DSNT of the spatial transformation coordinates, so that it can be differentiated from the output end to the input end, and the weight learning can be updated by back propagation;

[0038] A segmentation judgment unit is used to perform image segmentation judgment. If the image needs to be segmented, the target area and the background in the image are distinguished and segmented by MASK, that is, the target area and the non-target area are masked and classified;

[0039] The center point prediction unit is used to predict the center point of the document based on the heatmap to obtain the center point coordinate information. The center point prediction is auxiliary information used to improve the object detection accuracy.

[0040] As a further solution of the present invention: the key point prediction unit includes a loss function:

[0041] The loss function is expressed as:

[0042] in The output heatmap used to characterize the network SSH has a shape of Nx64x48x32.

[0043] The p is used to represent the coordinates of the four corner points of the document, expressed as: p(x|y);

[0044] The λ is used to characterize the weight, indicating L euc The impact of this model on the weights learned;

[0045] Said Used to characterize the coordinates of the four corner points predicted after dsnt processing, expressed as μ;

[0046] The L euc Used to represent the Euclidean loss function, used for regression coordinates, formula: L euc (μ|p)=||μ―p||2, directly calculate the two-dimensional Euclidean distance between μ and p;

[0047] The L reg Used to characterize the regularity constraint of the distribution, and to constrain the generated heat map to be closer to the Gaussian distribution. The formula is: Where d(·||·) represents the KL divergence measurement, which is widely used to measure the similarity between two distributions. represents a bivariate Gaussian distribution, p represents the coordinates of the corner point, Standard deviation is the hyperparameter for generating Gauss heatmap, in pixels.

[0048] As a further solution of the present invention: the segmentation judgment unit includes a loss function:

[0049] The loss function is:

[0050] Among them, the y i Used to represent the label of sample i, with positive class: 1, negative class: 0;

[0051] The p(y i ) is used to represent the probability of sample i being predicted as a positive class, which is the number after sigmoid;

[0052] The N is used to represent the number of samples, which here refers to the number of pixels in the image. After summing up, the average is taken.

[0053] As a further solution of the present invention: the center point prediction unit includes a loss function:

[0054] The loss function is: focal +λ1L size +λ2L off ,

[0055] Among them, the L focal :(1―p t ) γ log(p t ), p t Represents the predicted probability; γ is a hyperparameter whose control range is [0,5], which controls the contribution of easy-to-classify samples to the loss, thereby increasing the loss ratio of difficult-to-classify samples;

[0056] The L size :|p size ―gt size |, p size Indicates the height and width of the predicted document; gt size Indicates the height and width of the actual document; the height and width here are calculated based on the center point of the document;

[0057] Said Downsampling will produce float, which needs to be converted to Int. The difference between the two is Localoffset. Indicates the predicted local offset; p indicates the coordinates of the center point before scaling; R indicates the outputstride (output stride); Indicates the center point corresponding to downsampling, type is Int.

[0058] Compared with the prior art, the beneficial effects of the present invention are: by adopting a deep learning supervision method to fit the target scene, the data set source is a synthetic method, it has the ability to generalize to the real target scene, can effectively deal with the randomness of the text in the image and the complexity of the content, and efficiently and quickly locate and match the image document area that needs to be identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flowchart of a method for extracting documents from images.

[0060] Figure 2 A logical diagram of heatmap connected to fc (fully connected layer) in a method for extracting documents from images.

[0061] Figure 3 A method for extracting documents from images is used to extract heatmaps and obtain the corresponding maximum point by argmax.

[0062] Figure 4 A logical diagram of a method for extracting documents from images.

[0063] Figure 5 A schematic diagram of the architecture of a method for extracting documents from images DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0065] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0066] like Figure 1 and Figure 5 The method for extracting a document from an image provided by an embodiment of the present invention comprises the following steps:

[0067] S10, based on the feature pyramid network FPN, image scaling is performed at different layers to obtain multi-scale feature information;

[0068] S20, further enhancing feature extraction of the feature information by a single-stage detector SSH, wherein the single-stage detector SSH does not include a fully connected layer and has a high processing speed;

[0069] S30, based on the multi-head self-attention mechanism Multi-Head, performs multi-task joint optimization of network layers to optimize overfitting and generalization performance.

[0070] In this embodiment, a method for extracting documents from images is proposed. By adopting a deep learning supervision method to fit the target scene, the data set source is a synthetic method, which has the ability to generalize to the real target scene, can effectively deal with the randomness of the text in the image and the complexity of the content, and efficiently and quickly locate and match the image document area that needs to be identified; document extraction can be regarded as a key point detection task, that is, the position of the four corner points of the document in the image. The previous work is mainly as follows: heatmap is connected to fc (fully connected layer), and a representative example is RecursiveCNN: Figure 2 As shown in the figure, although the loss is differentiable from the input, the result is heavily dependent on the data distribution (for example, in the training set, an object always appears on the left side of the image; while in the test set, the object appears on the right side, which will lead to prediction errors). Secondly, the spatial information of the heatmap is lost through the fc conversion, so RecursiveCNN is a two-stage, and can only output a relatively rough coordinate point in the one-stage, which requires the subsequent Corner-Refiner to perform refined extraction. Therefore, it is not only time-consuming, but also heavily dependent on the data distribution of the training set; heatmap goes to argmax to get the corresponding maximum point: Figure 3 As shown, this method heatmap has spatial generalization, but because argmax is not differentiable in training, heatmap is usually used to approximate the encoded heatmap (Gaussian heatmap), resulting in inconsistency between the loss function and the final evaluation index; secondly, in the inference stage, only the coordinate point of the maximum response is used to calculate the numerical coordinate, while in the training stage, all coordinate points contribute to the loss; thirdly, there is a theoretical lower limit of error when converting heatmap into numerical coordinates; in the present invention, the advantages of both are combined to update the structure, such as Figure 4 As shown in the figure, we introduce DSNT (differentiable spatial to numerical transform) to take into account the advantages of the above solutions, obtain numerical coordinates in a differentiable way, and integrate the characteristics of components such as FPN and SSH as well as multi-task learning to build our entire model structure.

[0071] As another preferred embodiment of the present invention, the step of performing multi-task joint optimization of the network layer based on the multi-head self-attention mechanism Multi-Head specifically includes a specific technical solution:

[0072] The DSNT of spatial transformation coordinates enables the model to directly predict the position of key points, so that it can be differentiated from the output end to the input end, and the weight learning can be updated by back propagation;

[0073] Perform image segmentation judgment. If the image needs to be segmented, the target area and the background in the image are distinguished and segmented through MASK, that is, the target area and non-target area are masked and classified;

[0074] The center point of the document is predicted based on the heatmap to obtain the center point coordinate information. The center point prediction is auxiliary information used to improve the accuracy of object detection.

[0075] In this embodiment, the model consists of three components: FPN->SSH->Multi-Head. The output of FPN integrates feature information of different scales and is given to the SSH module for further feature enhancement. Finally, it is input into different tasks in Multi-Head respectively, Dsnt is the main task, and the rest are auxiliary tasks. Auxiliary tasks are conducive to improving model indicators. The following is a detailed description of each module, among which, FPN (Feature Pyramid Network): MobilenetV3 is used as the skeleton, and x4 x8x16 x32 (indicates the image reduction factor, for example, x4 means that the input image size (384x256) is reduced by 4 times to 96x64) is obtained at different layers. The advantage of this architecture is that it ensures the lightweight of the model and integrates the target recognition features of scenes of different scales (in some scenes, the detected targets will be smaller), which improves the robustness of the model in target detection at different scales; SSH: The full name is Single Stage Head Less. The name implies that the Head is removed. For example, VGG16 removes the fully-connected layers (fully connected layers: containing a large number of parameters). Single Stage refers to removing the fully connected layers on a network. Inserting SSH can achieve better performance than the previous Two There are two networks in Stage. SSH further strengthens the feature extraction of the feature layer. SSH uses three parallel convolutions: the first is 3x3 convolution, the second is two 3x3 convolutions instead of 5x5 convolutions, and the third is three 3x3 convolutions instead of 5x5 convolutions. Multi-Head: Multi-task and shared network parameters are an optimization method. Multiple tasks jointly optimize the parameters of the network layer. This approach can improve overfitting, generalization performance, and quickly learn through auxiliary information to obtain more efficient and compact data representation.

[0076] As another preferred embodiment of the present invention, the DSNT includes a loss function:

[0077] The loss function is expressed as:

[0078] in The output heatmap used to characterize the network SSH has a shape of Nx64x48x32.

[0079] The p is used to represent the coordinates of the four corner points of the document, indicating:

[0080] The λ is used to characterize the weight, indicating L euc The impact of this model on the weights learned;

[0081] Said Used to characterize the coordinates of the four corner points predicted after dsnt processing, expressed as μ;

[0082] The L euc Used to represent the Euclidean loss function, used for regression coordinates, formula: L euc (μ|p)=||μ―p||2, directly calculate the two-dimensional Euclidean distance between μ and p;

[0083] The L reg Used to characterize the regularity constraint of the distribution, and to constrain the generated heat map to be closer to the Gaussian distribution. The formula is: Where D(·||·) represents the KL divergence measurement, which is widely used to measure the similarity between two distributions. represents a bivariate Gaussian distribution, p represents the coordinates of the corner point, Standard deviation is the hyperparameter for generating Gauss heatmap, in pixels.

[0084] Furthermore, the MASK includes a loss function:

[0085] The loss function is:

[0086] Among them, the y i Used to represent the label of sample i, with positive class: 1, negative class: 0;

[0087] The p(y i ) is used to represent the probability of sample i being predicted as a positive class, which is the number after sigmoid;

[0088] The N is used to represent the number of samples, which here refers to the number of pixels in the image. After summing up, the average is taken.

[0089] Furthermore, the document center prediction includes a loss function:

[0090] The loss function is: focal +λ1L size +λ2L off ,

[0091] Among them, the L focal :(1―p t ) γ log(p t ), p t Represents the predicted probability; γ is a hyperparameter whose control range is [0,5], which controls the contribution of easy-to-classify samples to the loss, thereby increasing the loss ratio of difficult-to-classify samples;

[0092] The L size :|p size ―gt size |, p size Indicates the height and width of the predicted document; gt size Indicates the height and width of the actual document; the height and width here are calculated based on the center point of the document;

[0093] Said Downsampling will produce float, which needs to be converted to Int. The difference between the two is Localoffset. Indicates the predicted local offset; p indicates the coordinates of the center point before scaling; R indicates the outputstride (output stride); Indicates the center point corresponding to downsampling, type is Int.

[0094] In this embodiment, for the L focal :(1―p t ) γ log(p t ), explained as follows: When p t Tends to 1, which means that the sample is easy to distinguish. At this time, the modulation factor (1-p t ) γ It tends to 0, indicating that the contribution to the loss is small, that is, the loss ratio of easily distinguishable samples is reduced. When pt is very small, that is, if a sample is classified as a positive sample, but the probability that the sample is a foreground is very small, that is, it is misclassified as a positive sample, then the modulation factor (1-p t ) γ It tends to 1 and has little impact on the loss, so this loss function mainly focuses on the case of difficult-to-classify samples.

[0095] The present invention also provides a model for extracting documents from images, which comprises:

[0096] The multi-scale value module is used to perform image scaling and value taking at different levels based on the feature pyramid network FPN to obtain multi-scale feature information;

[0097] A feature enhancement module, used for further enhancing feature extraction of the feature information through a single-stage detector SSH, wherein the single-stage detector SSH does not include a fully connected layer and has an efficient processing speed;

[0098] The optimization fitting module is used to jointly optimize the network layer for multiple tasks based on the multi-head self-attention mechanism Multi-Head to optimize overfitting and generalization performance.

[0099] As another preferred embodiment of the present invention, the optimization fitting module includes:

[0100] The key point prediction unit is used to enable the model to directly predict the key point position through the DSNT of the spatial transformation coordinates, so that it can be differentiated from the output end to the input end, and the weight learning can be updated by back propagation;

[0101] A segmentation judgment unit is used to perform image segmentation judgment. If the image needs to be segmented, the target area and the background in the image are distinguished and segmented by MASK, that is, the target area and the non-target area are masked and classified;

[0102] The center point prediction unit is used to predict the center point of the document based on the heatmap to obtain the center point coordinate information. The center point prediction is auxiliary information used to improve the object detection accuracy.

[0103] As another preferred embodiment of the present invention, the key point prediction unit includes a loss function:

[0104] The loss function is expressed as:

[0105] in The output heatmap used to characterize the network SSH has a shape of Nx64x48x32.

[0106] The p is used to represent the coordinates of the four corner points of the document, expressed as: p(x|y);

[0107] The λ is used to characterize the weight, indicating L euc The impact of this model on the weights learned;

[0108] Said Used to characterize the coordinates of the four corner points predicted after dsnt processing, expressed as μ;

[0109] The L eucUsed to represent the Euclidean loss function, used for regression coordinates, formula: L euc (μ|p)=||μ―p||2, directly calculate the two-dimensional Euclidean distance between μ and p;

[0110] The L reg Used to characterize the regularity constraint of the distribution, and to constrain the generated heat map to be closer to the Gaussian distribution. The formula is: Where D(·||·) represents the KL divergence measurement, which is widely used to measure the similarity between two distributions. represents a bivariate Gaussian distribution, p represents the coordinates of the corner point, Standard deviation is the hyperparameter for generating Gauss heatmap, in pixels.

[0111] As another preferred embodiment of the present invention, the segmentation judgment unit includes a loss function:

[0112] The loss function is:

[0113] Among them, the y i Used to represent the label of sample i, with positive class: 1, negative class: 0;

[0114] The p(y i ) is used to represent the probability of sample i being predicted as a positive class, which is the number after sigmoid;

[0115] The N is used to represent the number of samples, which here refers to the number of pixels in the image. After summing up, the average is taken.

[0116] As another preferred embodiment of the present invention, the center point prediction unit includes a loss function:

[0117] The loss function is: focal +λ1L size +λ2L off ,

[0118] Among them, the L focal :(1―p t ) γ log(p t ), p t Represents the predicted probability; γ is a hyperparameter whose control range is [0,5], which controls the contribution of easy-to-classify samples to the loss, thereby increasing the loss ratio of difficult-to-classify samples;

[0119] The L size :|p size ―gt size |, p size Indicates the height and width of the predicted document; gtsize Indicates the height and width of the actual document; the height and width here are calculated based on the center point of the document;

[0120] Said Downsampling will produce float, which needs to be converted to Int. The difference between the two is Localoffset. Indicates the predicted local offset; p indicates the coordinates of the center point before scaling; R indicates the outputstride (output stride); Indicates the center point corresponding to downsampling, type is Int.

[0121] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0122] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the disclosure in the specification and examples. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0123] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for extracting a document from an image, characterized in that: Include: Based on the feature pyramid network FPN, image scaling is performed at different layers to obtain multi-scale feature information; Further enhancing feature extraction of the feature information by a single-stage detector SSH, wherein the single-stage detector SSH does not include a fully connected layer and has an efficient processing speed; It can jointly optimize the network layer through multiple tasks based on the multi-head self-attention mechanism Multi-Head to optimize overfitting and generalization performance.

2. A method for extracting documents from images according to claim 1, characterized in that: The step of jointly optimizing the network layer by multiple tasks based on the multi-head self-attention mechanism Multi-Head specifically includes the following specific technical solutions: The DSNT of spatial transformation coordinates enables the model to directly predict the position of key points, so that it can be differentiated from the output end to the input end, and the weight learning can be updated by back propagation; Perform image segmentation judgment. If the image needs to be segmented, the target area and the background in the image are distinguished and segmented through MASK, that is, the target area and non-target area are masked and classified; The center point of the document is predicted based on the heatmap to obtain the center point coordinate information. The center point prediction is auxiliary information used to improve the accuracy of object detection.

3. A method for extracting documents from images according to claim 2, characterized in that: The DSNT contains the loss function: The loss function is expressed as: in The output heatmap used to characterize the network SSH has a shape of Nx64x48x32. The p is used to represent the coordinates of the four corner points of the document, indicating: The λ is used to characterize the weight, indicating L euc The impact of this model on the weights learned; Said Used to characterize the coordinates of the four corner points predicted after dsnt processing, expressed as μ; The L euc Used to characterize the Euclidean loss function, used for regression coordinates, formula: Directly calculate the two-dimensional Euclidean distance between μ and p; The L reg It is used to characterize the regularity constraint of the distribution and constrain the generated heat map to be closer to the Gaussian distribution. The formula is: Where D(·∥·) represents the KL divergence measurement, which is widely used to measure the similarity between two distributions. represents a bivariate Gaussian distribution, p represents the coordinates of the corner point, Standard deviation is the hyperparameter for generating gauss heatmap, in pixels.

4. A method for extracting documents from images according to claim 3, characterized in that: The MASK includes the loss function: The loss function is: Among them, the y i Used to represent the label of sample i, with positive class: 1, negative class: 0; The p(y i ) is used to represent the probability of sample i being predicted as a positive class, which is the number after sigmoid; The N is used to represent the number of samples, which here refers to the number of pixels in the image. After summing, the average is taken.

5. A method for extracting documents from images according to claim 4, characterized in that: The document center prediction includes the loss function: The loss function is: focal +λ1L size +λ2L off , Among them, the L focal :(1―p t ) γ log(p t ), p t Represents the predicted probability; γ is a hyperparameter whose control range is [0,5], which controls the contribution of easy-to-classify samples to the loss, thereby increasing the loss ratio of difficult-to-classify samples; The L size :|p size ―gt size |, p size Indicates the height and width of the predicted document; gt size Indicates the height and width of the actual document; the height and width here are calculated based on the center point of the document; Said Downsampling will produce float, which needs to be converted to Int. The difference between the two is Localoffset. Indicates the predicted local offset; p indicates the coordinates of the center point before scaling; R indicates the outputstride (output stride); Indicates the center point corresponding to downsampling, type is Int.

6. A method and model for extracting documents from images, characterized in that: Include: The key point prediction unit is used to enable the model to directly predict the key point position through the DSNT of the spatial transformation coordinates, so that it can be differentiated from the output end to the input end, and the weight learning can be updated by back propagation; A segmentation judgment unit is used to perform image segmentation judgment. If the image needs to be segmented, the target area and the background in the image are distinguished and segmented by MASK, that is, the target area and the non-target area are masked and classified; The center point prediction unit is used to predict the center point of the document based on the heatmap to obtain the center point coordinate information. The center point prediction is auxiliary information used to improve the object detection accuracy.

7. A method and model for extracting documents from images according to claim 6, characterized in that: The optimization fitting module includes: The key point prediction unit is used to enable the model to directly predict the key point position through the DSNT of the spatial transformation coordinates, so that it can be differentiated from the output end to the input end, and the weight learning can be updated by back propagation; A segmentation judgment unit is used to perform image segmentation judgment. If the image needs to be segmented, the target area and the background in the image are distinguished and segmented by MASK, that is, the target area and the non-target area are masked and classified; The center point prediction unit is used to predict the center point of the document based on the heatmap to obtain the center point coordinate information. The center point prediction is auxiliary information used to improve the object detection accuracy.

8. A method and model for extracting documents from images according to claim 7, characterized in that: The key point prediction unit includes a loss function: The loss function is expressed as: in The output heatmap used to characterize the network SSH has a shape of Nx64x48x32. The p is used to represent the coordinates of the four corner points of the document, indicating: The λ is used to characterize the weight, indicating L euc The impact of this model on the weights learned; Said Used to characterize the coordinates of the four corner points predicted after dsnt processing, expressed as μ; The L euc Used to characterize the Euclidean loss function, used for regression coordinates, formula: Directly calculate the two-dimensional Euclidean distance between μ and p; The L reg It is used to characterize the regularity constraint of the distribution and constrain the generated heat map to be closer to the Gaussian distribution. The formula is: Where D(·∥@) represents the KL divergence measurement, which is widely used to measure the similarity between two distributions. represents a bivariate Gaussian distribution, p represents the coordinates of the corner point, Standard deviation is the hyperparameter for generating gauss heatmap, in pixels.

9. A method and model for extracting documents from images according to claim 8, characterized in that: The segmentation judgment unit includes a loss function: The loss function is: Among them, the y i Used to represent the label of sample i, with positive class: 1, negative class: 0; The p(y i ) is used to represent the probability of sample i being predicted as a positive class, which is the number after sigmoid; The N is used to represent the number of samples, which here refers to the number of pixels in the image. After summing up, the average is taken.

10. A method and model for extracting documents from images according to claim 9, characterized in that: The center point prediction unit includes a loss function: The loss function is: focal +λ1L size +λ2L off , Among them, the L focal :(1―p t ) γ log(p t ), p t Represents the predicted probability; γ is a hyperparameter whose control range is [0,5], which controls the contribution of easy-to-classify samples to the loss, thereby increasing the loss ratio of difficult-to-classify samples; The L size :|p size ―gt size |, p size Indicates the height and width of the predicted document; gt size Indicates the height and width of the actual document; the height and width here are calculated based on the center point of the document; Said Downsampling will produce float, which needs to be converted to Int. The difference between the two is Localoffset. Indicates the predicted local offset; p indicates the coordinates of the center point before scaling; R indicates the outputstride (output stride); Indicates the center point corresponding to downsampling, type is Int.

Images