Document image processing method and device, electronic device, and storage medium

Abstract

The disclosed embodiments relate to a document image processing method and apparatus, electronic device, and storage medium, and relate to the field of image processing technology. The document image processing method includes: obtaining a document image to be processed, performing document edge detection on the document image to obtain document edges; performing straight line fitting on the document edges to determine a set of straight lines; determining four vertices of the document in the document image to be processed based on the set of straight lines; and performing a fill operation and perspective transformation on the document image to be processed based on the four vertices to obtain a document correction result. The disclosed technical solution can improve the accuracy of document correction.

Description

[0001] The present disclosure relates to the field of image processing technology, and in particular to a document image processing method, a document image processing device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the popularity of smartphones and the improvement of camera quality, users can now take photos of documents with their phones to obtain copies. However, due to the difficulty in precisely controlling the shooting angle and framing range, the document images taken by users may suffer from perspective distortion and invalid background areas.

[0003] In order to solve the above technical problems, the document area can be determined by performing image foreground and background segmentation and fitting some boundary lines, or the straight lines can be divided into horizontal lines and vertical lines for line detection, and different methods can be used to determine the contour boundaries for different types.

[0004] The above method uses image foreground and background segmentation to determine the document area and fit boundary lines. This involves many steps, resulting in low efficiency, reliability, and accuracy. Classifying lines as horizontal or vertical for line detection and using different methods to determine contour boundaries for different types of lines also has certain limitations and a limited scope of application.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0006] The purpose of the present disclosure is to provide a document image processing method and device, electronic device, and storage medium, thereby overcoming, at least to a certain extent, the problem of poor document image correction results generated due to limitations and defects of related technologies.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a document image processing method is provided, characterized in that it includes: obtaining a document image to be processed, and performing document edge detection on the document image to be processed to obtain a document edge; performing straight line fitting on the document edge to determine a straight line set; determining four vertices of the document in the document image to be processed based on the straight line set; and performing a filling operation and a perspective transformation on the document image to be processed in combination with the four vertices to obtain a document correction result.

[0009] According to one aspect of the present disclosure, a document image processing device is provided, comprising: an edge acquisition module for acquiring a document image to be processed and performing document edge detection on the document image to be processed to obtain a document edge; a straight line set determination module for performing straight line fitting on the document edge to determine a straight line set; a document vertex positioning module for determining four vertices of the document in the original document image to be processed based on the straight line set; and an image correction module for performing a filling operation and a perspective transformation on the document image to be processed in combination with the four vertices to obtain a document correction result.

[0010] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above-mentioned document image processing methods by executing the executable instructions.

[0011] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the document image processing method described above is implemented.

[0012] In the document image processing method, document image processing device, electronic device, and computer-readable storage medium provided in the embodiments of the present disclosure, on the one hand, by combining pixel and semantic dimensions to perform document edge detection on a document image to be processed, the document edge is obtained. This allows for global consistency of the document edge, eliminates interference from erroneous edges in the background and within the image, improves the accuracy and precision of edge detection, reduces the number of steps, and enhances operational efficiency and reliability. On the other hand, a straight line fitting is performed on the document edge to determine a set of straight lines. Four vertices of the document in the document image to be processed are then determined based on the set of straight lines. A fill operation is then performed on the document image to perform a perspective transformation, thereby obtaining a document correction result. The ability to perform a perspective transformation on the document image based on the four vertices improves the reliability of the straight line fitting and accurately locates the vertices. Furthermore, the fill operation ensures that the corrected image remains level, avoiding corner loss and limitations. This allows for accurate correction of the document image to be processed, improving the accuracy of the document correction result. Furthermore, accurate correction can be performed on all types of document images or those of varying sizes, improving the accuracy and versatility of image correction and expanding its scope of application.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0015] Figure 1 A schematic diagram showing a system architecture to which the image processing method or image processing apparatus according to the embodiments of the present disclosure can be applied.

[0016] Figure 2 A schematic diagram schematically illustrates a document image processing method in an embodiment of the present disclosure.

[0017] Figure 3 The following schematically illustrates a flow chart of determining document edges in an embodiment of the present disclosure.

[0018] Figure 4 The structural diagram of the edge detection model in the embodiment of the present disclosure is schematically shown.

[0019] Figure 5 A flowchart schematically illustrates the impact of the loss function on the edge image in an embodiment of the present disclosure.

[0020] Figure 6 A schematic diagram schematically illustrates how to obtain document edges by edge detection in an embodiment of the present disclosure.

[0021] Figure 7 The flowchart of the straight line fitting in the embodiment of the present disclosure is schematically shown.

[0022] Figure 8 The result diagram of straight line merging in the embodiment of the present disclosure is schematically shown.

[0023] Figure 9 An image diagram schematically illustrating straight line fitting in an embodiment of the present disclosure is shown.

[0024] Figure 10 The following schematically illustrates a flow chart of determining vertices in an embodiment of the present disclosure.

[0025] Figure 11 A schematic diagram schematically illustrates four vertices determined in an embodiment of the present disclosure.

[0026] Figure 12 The flowchart of filtering vertices in the embodiment of the present disclosure is schematically shown.

[0027] Figure 13 The following schematically illustrates a process flow diagram for performing perspective transformation in an embodiment of the present disclosure.

[0028] Figure 14 A schematic diagram schematically illustrates image processing without performing a filling operation in an embodiment of the present disclosure.

[0029] Figure 15 A schematic diagram schematically illustrates image processing requiring a filling operation in an embodiment of the present disclosure.

[0030] Figure 16 A schematic diagram schematically illustrates the overall process of image correction in an embodiment of the present disclosure.

[0031] Figure 17 A block diagram schematically shows a document image processing device in an embodiment of the present disclosure.

[0032] Figure 18 A schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0034] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] Because it's difficult to precisely control the shooting angle and framing range, document images captured by users often suffer from two issues: The document is skewed in the image, resulting in perspective distortion; and the document often only occupies the center of the image, leaving a significant amount of ineffective background surrounding it. These issues make reading difficult and unsuitable for archiving. Furthermore, directly performing OCR on such document images often results in poor text recognition. Therefore, it's necessary to correct the perspective of the captured document image, converting it to a frontal view, and then cropping out the document area to address these issues.

[0036] Related technologies offer the following approaches: Using the traditional Canny operator or GPB algorithm for image foreground and background segmentation (document and background region division) has poor robustness and is not suitable for images with complex backgrounds. It requires traversing all possible polygons (quadrilaterals) formed by all lines to determine the final document region, a cumbersome and computationally intensive process. During line detection, lines are classified as either horizontal or vertical, which is not applicable for situations with severe perspective distortion or when the document is rotated approximately 45 degrees within the image and has certain limitations. Line sets are divided into horizontal and vertical categories, and all intersections of any two lines are traversed to obtain a polygon set. The polygon with the largest area is then selected as the document region. Using a Gaussian mixture model for foreground and background segmentation has poor robustness. When using contours for line detection, only four possible document boundary lines are fitted, which can easily lead to offsets or tilts, making it difficult to fit the actual document boundary. Using Canny for document edge detection is highly dependent on parameter settings, resulting in poor robustness and generalization. This approach skips edge detection and instead directly performs line detection on the document image's grayscale image. This approach struggles with complex backgrounds or unclear document boundaries. Different methods are used for obtaining the four contour lines (top, bottom, left, and right) for different document types (text, tables, and graphics), resulting in limited versatility.

[0037] In order to solve the above technical problems, an image processing method is provided in an embodiment of the present disclosure, which can be applied to various types of document image recognition scenarios. Figure 1 A schematic diagram showing a system architecture to which the image processing method or image processing apparatus according to the embodiments of the present disclosure can be applied.

[0038] like Figure 1As shown, the system architecture 100 may include: a first end 110, a network 120, and a second end 130. The first end 110 may be any type of device capable of displaying images, such as a computer, a smartphone, a tablet computer, a smart wearable device (such as AR glasses), a robot, a drone, or other device capable of capturing images. The first end 110 may establish a communication connection with the second end 130 via the network 120, and transmit the collected document image to be processed to the second end 130, which then performs image processing. The second end may be a smart device, such as a printer, a scanner, or a computer, or may be a server, etc. Figure 1 The first terminal 110 and the second terminal 130 are different devices. In one embodiment, the first terminal 110 and the second terminal 130 may be the same device, that is, both the first terminal and the second terminal are smart devices, such as smart phones.

[0039] It should be noted that in this exemplary embodiment, the image uploaded by the first end 110 is corrected via the second end 130 . If the first end and the second end are the same, this is equivalent to performing document correction on the document image to be processed via the first end.

[0040] It should be understood that Figure 1 The number of devices is merely illustrative. For example, according to implementation needs, multiple first ends can be provided, each of which is connected to the second end 130 via the network 120. The second end 130 can simultaneously perform document correction on each document image to be processed at the first end, and so on.

[0041] In order to solve the problems in the related art, a document image processing method is provided in the embodiment of the present disclosure. Figure 2 The document image processing method in the embodiment of the present disclosure is described in detail.

[0042] In step S210 , a document image to be processed is acquired, and document edge detection is performed on the document image to be processed to obtain document edges.

[0043] In the disclosed embodiments, the document image to be processed may be images of various types and sizes containing documents, including, but not limited to, books (covers / inside pages), cards (business cards / ID cards / bank cards), bills (train tickets / air tickets / invoices / receipts / receipts, etc.), paper (exam papers / posters / menus / flyers / newspapers), PowerPoint projections / TVs / computer screens, etc. The document image to be processed may be tilted or placed in any orientation.

[0044] Edge detection is used to determine the boundaries of a document image to be processed. In the disclosed embodiment, document edge detection can be performed in a combination of pixel and semantic dimensions to improve accuracy. Figure 3The specific flow chart for determining the document edge is shown schematically in FIG. Figure 3 As shown in , it mainly includes the following steps:

[0045] In step S310, document edge detection is performed on the document image to be processed using an edge detection model to obtain a first document edge.

[0046] In this step, the edge detection model DocEdgeNet is a deep learning model based on the extremely lightweight image segmentation network SINet. The edge detection model DocEdgeNet is an encoding-decoding type pixel-level dense prediction network. Figure 4 The network structure of the edge detection model is schematically shown in FIG. Figure 4 As shown in , the edge detection model mainly includes an encoder part 401 and a decoder part 402, and the edge detection model provides a semantic segmentation branch and an edge detection branch. By fusing the features of the semantic segmentation branch with the features of the edge detection branch, more supervision information is provided for edge detection, which can largely eliminate edge interference from the background and the document itself, thereby improving the accuracy of document edge detection.

[0047] Among them, the edge detection model specifically includes feature blocks of multiple modules, for example, it can include a feature block 410 represented by two-dimensional convolution Conv2D, a feature block 420 of the S2 module S2-module, a feature block 430 of the upsampling module, a feature block 440 of the DSConv+SE module, and a dimensionality reduction convolution feature block 450, and a dimensionality reduction convolution + upsampling feature block 460.

[0048] The S2-module follows a split-transform-merge strategy. It first uses group pointwise convolution to halve the number of feature maps. It also performs a channel shuffle operation to enable information exchange between feature maps from different groups. The halved feature maps are then input into two S2-blocks, and the outputs of each S2-block are merged via concat fusion. Residual connections are also used in the S2-module. Average pooling is used to implement a multi-level receptive field structure in the S2-block because a larger hole rate increases inference latency. Furthermore, the multi-path structure cannot support GPU parallel computing. After average pooling, depthwise separable convolution is used. Empirically, performing pointwise conv before or after bilinear upsampling has no decisive impact on accuracy. Therefore, placing pointwise conv before upsampling can reduce computational complexity.

[0049] Continue to refer Figure 4As shown in , out_seg is the result of the semantic segmentation branch, i.e., the mask; out_edge is the prediction result of the edge detection branch, i.e., the edge. gt_seg is the label of semantic segmentation, and gt_edge is the label of edge detection. Based on this, the semantic segmentation loss L can be calculated. s and edge detection loss L e . The network model parameters are further updated according to the loss function to minimize the loss function and complete the training process.

[0050] It should be noted that the deep learning model can be trained to obtain an edge detection model. A large amount of training data is used to train the model, a small part of which is real data and manually annotated, and the vast majority is synthetic data. Real data is generally actually taken by a mobile phone and manually annotated. The present disclosure uses a synthetic method to generate a large amount of training data. It only requires repeated random combinations of several foreground pictures and several background pictures to generate training data. Among them, before each combination, the foreground picture needs to be randomly perspective transformed and the background picture needs to be randomly cropped to ensure that the synthetic picture has diversity.

[0051] Specifically, the loss function can be determined based on the edge perception loss and the semantic segmentation loss, and the model parameters of the deep learning model can be adjusted with the minimum loss function as the training goal to achieve model training, thereby obtaining an edge detection model. Specifically, the loss function can be determined according to formula (1):

[0052] L=L e +λL s Formula (1)

[0053] Among them L e is the edge-aware loss, L s is the semantic segmentation loss, λ is L s The adjustment factor of L is tested experimentally and the model accuracy is the highest when λ=0.5. e The essence of is the weighted cross entropy loss, which can be used to solve the problem of imbalance between the two categories of "edge pixels" and "non-edge pixels" and to supervise the boundaries of documents. s For the edge detection model DocEdgeNet, it is an auxiliary loss used to supervise the document area and can embed richer semantic information into the feature map, which is conducive to maintaining global consistency, thereby eliminating the interference of erroneous edges in the background and within the document, and improving the precision and accuracy of edge detection.

[0054] exist Figure 4Based on the network structure shown in , in order to obtain the boundaries of the document image, DocEdgeNet obtains feature maps of different resolutions from the encoder and decoder respectively. For example, the encoder obtains a feature map with a first resolution, and the decoder obtains a feature map with a second resolution. After obtaining the feature maps, these feature maps can be subjected to dimensionality reduction convolution pointwise conv to obtain the processing results. Dimensionality reduction convolution PW refers to 1*1 convolution. Through dimensionality reduction convolution, the output dimension can be changed, and the number of parameters and calculations can be reduced. The convolution results are then upsampled to obtain upsampled feature maps, so that the upsampled feature maps have the same width and height. The upsampled feature maps are further merged and concat to achieve feature fusion, and finally the document edge is predicted based on the fused features.

[0055] refer to Figure 5 As shown in Figure 5 For the document image of the bill type shown in Figure A, Figure 5 Figure B does not use semantic segmentation loss L s The prediction results of the trained model, Figure 5 Figure C in the figure uses semantic segmentation loss L s The prediction results of the trained model. By comparison, we can see that using semantic segmentation loss L s It can effectively suppress erroneous edges in the background or inside the document area, significantly improving the accuracy and quality of document edge detection.

[0056] Because the edge detection model is trained by combining pixels and semantics, the document image to be processed is input into the edge detection model for edge prediction, a document edge probability map is obtained, and the document edge probability map is determined as the first document edge. In the document edge probability map, the value of each pixel represents the probability that the pixel at the corresponding position in the document image to be processed belongs to the document edge, and the value range is [0.0, 1.0]. The larger the value, the higher the probability that the pixel belongs to the document edge. When visualizing the edge probability, [0.0, 1.0] can be mapped to [0, 255], which is a grayscale image. The whiter the color, the greater the probability of belonging to the document edge. In the disclosed embodiment, document edge detection is performed by combining pixels and semantics, avoiding the problems of poor model robustness, poor generalization ability, and strong dependence on parameters in related technologies. It can solve the problem of imbalance between edge pixel and non-edge pixel categories, and can effectively suppress the interference of false edges in the background or document area, improve the accuracy of edge detection, and improve the comprehensiveness of the first document edge, thereby improving robustness and reliability. It can be applied to edge detection in various situations, expanding the scope of application. In addition, the edge detection model DocEdgeNet is a lightweight model with a model weight of only 360K and a floating-point operation number of 0.085M FLOPs used to evaluate the model's computational complexity. Coupled with the high efficiency of the subsequent processing process, it can run in real time on mobile phones, improving operational efficiency.

[0057] In step S320, the first document edge is binarized to obtain a second document edge.

[0058] In this step, the binarization process refers to the threshold binarization process. The first document edge is binarized using the threshold binarization method to obtain the second document edge. The specific process is to traverse the edge probability value of each pixel in the document edge probability map represented by the first document edge. If the probability value of a certain pixel is greater than or equal to the set threshold p (for example, 0.85), then the edge probability of the pixel is reassigned to 1. If the probability value of a certain pixel is less than the set threshold value, then the edge probability of the pixel is reassigned to 0. Similarly, traverse the visualized document edge probability map. If the value of the pixel is greater than 255×p, then the pixel is assigned to 255, otherwise it is assigned to 0, thereby obtaining a document edge binary map, which is recorded as the second document edge.

[0059] In step S330, the second document edge is filtered to obtain a third document edge.

[0060] In this step, the second document edge may contain some edge blocks in the background or within the document area. These edge blocks are not document edges but rather interference items, so they need to be filtered out to improve accuracy. The process of filtering the second document edge includes: using a connected component algorithm to find all connected edges in the second document edge and calculating the area of ​​each connected edge. If the area is less than a set threshold, it is filtered out, and only connected edges with an area greater than the predetermined threshold are retained. Based on this, the third document edge is obtained by filtering the second document edge.

[0061] In step S340, the third document edge is thinned to obtain a fourth document edge as the document edge.

[0062] In this step, the skeleton of the image can be obtained through the refinement algorithm. The refinement algorithm here can be the ZhangSuen refinement algorithm. The specific process includes: Step 1, looping all foreground pixels, marking the pixels that meet the first condition as deleted, the first condition includes: 1.2<=N(p1)<=6; 2.S(P1)=1; 3.P2*P4*P6=0; 4.P4*P6*P8=0. Among them, N(p1) represents the number of foreground pixels among the 8 pixels adjacent to P1. S(P1) represents the cumulative number of times 0~1 appears from P2~P9~P2 pixels, where 0 represents background and 1 represents foreground.

[0063] Step 2: Pixels P1 that meet the second condition are marked for deletion. The second condition is as follows: 1. 2 <= N(p1) <= 6; 2. S(p1) = 1; 3. P2*P4*P8 = 0; 4. P2*P6*P8 = 0. Repeat these two steps until no pixels are marked for deletion in either step. The output is the skeleton after thinning the binary image.

[0064] In the third document edge, the edge thickness is usually greater than one pixel. Direct line detection would produce many invalid lines, so it is necessary to refine the edge. The ZhangSuen thinning algorithm is used to extract the edge skeleton, obtaining the fourth document edge, which is then determined as the document edge.

[0065] Figure 6 The schematic diagram of edge detection is shown in FIG. Figure 6 As shown in , it mainly includes: obtaining the document image to be processed, such as Figure 6 As shown in Figure A in FIG; the document image to be processed is subjected to document edge detection to obtain a document edge probability map, that is, the first document edge, as shown Figure 6 As shown in Figure B in FIG; Binarization processing is performed on the edge of the first document to obtain the edge of the second document, as shown in FIG. Figure 6As shown in Figure C in FIG; the second document edge is filtered to obtain the third document edge, such as Figure 6 As shown in FIG. D in FIG. ; the third document edge is thinned to obtain a fourth document edge as the document edge, as shown in FIG. Figure 6 As shown in Figure E in .

[0066] In step S220, straight line fitting is performed on the document edge to determine a straight line set.

[0067] In the embodiment of the present disclosure, line fitting is used to determine a line, thereby determining the boundary of the document image to be processed. The line fitting may include the steps of line detection, line merging, and line filtering.

[0068] Figure 7 The flow chart for performing straight line fitting is schematically shown in FIG. Figure 7 As shown in , it mainly includes the following steps:

[0069] In step S710, line detection is performed on the edge of the document to determine a first line set.

[0070] In this step, the document edge represented by the fourth document edge can be detected using the probabilistic Hough transform to detect the lines contained therein. Assuming that n lines are detected in total and these lines are represented by two-point form, the set of all two-point line forms can be represented as L′={l′1,l′2,...,l′ n Furthermore, the two-point straight line set can be converted into a parametric straight line set L1 = {l1, l2, ..., l n}, and the parametric straight line set is recorded as the first straight line set. The two-point straight line is represented by two points on the straight line. If the two-point form of the straight line l is [(x1, y1), (x2, y2)], then its parametric form is (ρ, θ), where ρ is the distance from the origin of the image coordinate system to the straight line l. When l intersects the Y-axis at the negative semi-axis, the value is negative; when l intersects the Y-axis at the positive semi-axis, the value is positive. θ is the angle between the straight line l and the negative semi-axis of the Y-axis, in radians, and the range of the angle is [0, π]. The corresponding relationship between the distance ρ from the origin of the image coordinate system to the straight line and the angle θ between the straight line and the negative semi-axis of the Y-axis and the coordinates of the two points on the straight line is shown in formulas (2) and (3):

[0071]

[0072]

[0073] In step S720 , the lines in the first line set are clustered to determine line clusters for line merging, and a second line set is obtained based on the centers of the line clusters.

[0074] In this step, among the multiple straight lines in the first straight line set, some straight lines have high similarity, so it is necessary to merge the straight lines with high similarity to facilitate subsequent processing. The specific process of straight line merging is as follows: Figure 8 As shown in , the specific steps include:

[0075] In step S810, the straight lines in the first straight line set are sorted in ascending order according to the distance from the coordinate origin of the document image to be processed to the straight line.

[0076] In this step, the straight lines in the first straight line set are sorted in ascending order according to the distance ρ between the coordinate origin of the document image to be processed and the straight line l.

[0077] In step S820 , a first type of line cluster is determined based on the sorted first line.

[0078] In this step, the first type of line cluster c1 can be determined based on the first line (the line with the smallest distance) in the sorted first line set. Specifically, they can be initialized as c1 = {l1,}, C = {c1,}. Each type of line cluster c i It is used to store the same type of straight lines, that is, approximate straight lines, such as Figure 9 The three vertical straight lines on the left side of Figure C.

[0079] In step S830, the distance difference and angle difference between the next straight line and the center of the last type of straight line cluster are obtained, the next straight line is determined as the target type straight line cluster based on the distance difference and the angle difference, and a straight line cluster is obtained based on the first type of straight line cluster and the target type straight line cluster.

[0080] In this step, the next straight line can be any straight line in the first straight line set except the first straight line. The next straight line can be represented as l i (2≤i≤n). The last type of straight line cluster c in the first straight line set C can be calculated based on the next straight line -1 Center The difference in distance ρ and angle θ between the next line and the center of the last line cluster can be calculated. Based on the comparison of the distance difference and angle difference with a set threshold, a target line cluster is determined based on the next line. The set threshold corresponding to the distance is different from the set threshold corresponding to the angle. The target line cluster can be a newly created line cluster or the last line cluster. Specifically, if the distance difference and angle difference between the next line and the center of the last line cluster are both less than the corresponding set thresholds, the next line is added to the last line cluster. If either the distance difference or the angle difference between the next line and the center of the last line cluster is greater than the corresponding set threshold, a newly created line cluster is determined based on the next line and added to the line clusters. This step is repeated until all lines in the first line set are traversed, and a newly created line cluster or the last line cluster is determined. Based on this, a line cluster can be generated based on the first line cluster and the newly created line cluster or the last line cluster included in the target line cluster.

[0081] For example, if the next line l i and the center of the last type of line cluster If the distance difference and angle difference are both less than the set threshold, the next straight line l i Add to the last type of line cluster c -1 Otherwise, the following straight line l i Based on this, a new class of straight line cluster c is created. j ={l i ,}, and create a new class line cluster c j Add to the line cluster C; finally get C={c1,c2,...,c m}. Each c i There is at least one straight line in the equation. If there are multiple straight lines, then the multiple straight lines are approximately the same straight line.

[0082] In step S840 , the center of each type of line cluster in the line clusters is calculated, and a set consisting of the centers of each type of line cluster is determined as a line merging result to determine a second line set.

[0083] In this step, after completing the above straight line clustering, calculate each type of straight line cluster c i Center All line cluster centers The set of That is the straight line merging result, and the straight line merging result can be determined as the second straight line set.

[0084] In step S730 , the lines in the second line set are filtered according to the filtering condition to obtain a third line set as the line set.

[0085] In this step, after the lines are merged, there may be erroneous lines that are not on the document boundary. At this time, line filtering is required to improve the accuracy of the lines. The specific process of line filtering is: filtering according to the filtering conditions. The filtering conditions are used to indicate the range of the lines to be filtered. The filtering conditions are to filter the horizontal or vertical lines close to the image boundary in the second line set. After filtering, the filtered second line set is clustered using the union-find algorithm to obtain cluster categories. If the number of cluster categories is less than the number of categories, the previous filtering is canceled. The number of categories can be 4. The filtered line set is recorded as the third line set, and the number of lines in the third line set is less than or equal to the number of lines in the second line set.

[0086] Figure 9 The image of the straight line fitting is shown schematically in FIG. Figure 9 As shown in , first obtain the document image to be processed, such as Figure 9 As shown in Figure A in FIG; obtain the fourth document edge corresponding to the document image to be processed, for example Figure 9 As shown in Figure B in FIG; perform straight line detection on the edge of the fourth document to obtain a first straight line set, for example Figure 9 As shown in Figure C in ; cluster the straight lines in the first straight line set to obtain the second straight line set, as shown in Figure 9 As shown in Figure D in FIG; the straight lines in the second straight line set are filtered to obtain a third straight line set, so that the third straight line set is used as the straight line set, as shown in FIG. Figure 9 As shown in Figure E in .

[0087] Continue to refer Figure 2 As shown in , in step S230 , four vertices of the document in the document image to be processed are determined based on the set of straight lines.

[0088] In an embodiment of the present disclosure, after achieving document boundary straight line fitting, the document vertex positioning is performed using a set of straight lines represented by a third set of straight lines to determine the positions of the four vertices of the document contained in the document image to be processed. The vertex positioning process includes calculating the candidate intersection set, intersection clustering, intersection filtering and other processes.

[0089] Figure 10 The flowchart for determining the vertex is shown schematically in FIG. Figure 10 As shown in , it mainly includes the following steps:

[0090] In step S1010, the angle between each two straight lines in the straight line set is calculated to determine the intersection of each two straight lines, and a candidate intersection set is formed based on the intersection;

[0091] In step S1020, all intersection points in the candidate intersection point set are aggregated into multiple point clusters;

[0092] In step S1030, the multi-class point clusters are filtered, and the four vertices of the document in the document image to be processed are determined according to the centers of the filtered multi-class point clusters.

[0093] Specifically, for Figure 11 The document image to be processed shown in Figure A is first fitted with a straight line to obtain the following Figure 11 The third straight line set shown in Figure B is taken as the straight line fitting result. Next, the angle between two straight lines in the third straight line set is calculated. If the angle is within the range of (50°, 90°], the intersection of the two straight lines is calculated. All the intersections of two straight lines constitute the candidate intersection set. Figure 11 As shown in Figure C.

[0094] Furthermore, the K-Means algorithm is used to aggregate all the intersection points in the candidate intersection point set into multiple categories to form multi-class point clusters. Specifically, the points can be aggregated into four categories to obtain four types of clusters, which are recorded as C1, C2, C3, and C4.

[0095] Next, the multi-class point clusters can be filtered to obtain the four vertices of the document in the document image to be processed, such as Figure 11 As shown in Figure F in . The specific process of determining the vertex can be Figure 12 As shown in the reference Figure 12 As shown in , it mainly includes the following steps:

[0096] In step S1210 , the intersection points in the plurality of point clusters are divided into a first type of point cluster and a second type of point cluster according to the positional relationship between the positions of the intersection points and the range of the document image to be processed.

[0097] The first type of point cluster refers to the point cluster located within the image, and the second type of point cluster refers to the point cluster located outside the image. Specifically, the intersection points in the point cluster can be divided into the first type of point cluster C according to the positional relationship between the position of the intersection point and the range of the document image to be processed. i,inner and the second type of point cluster C i,outer , and the first type of point cluster and the second type of point cluster together form point cluster C i (1≤i≤4). Figure 11 As shown in Figure D in .

[0098] In step S1220, the first type of point clusters are filtered according to the existence status of the second type of point clusters.

[0099] The existence status of the second type of point cluster can be used to indicate whether the second type of point cluster is empty. Specifically, in step S1221, if the existence status indicates that the second type of point cluster is empty, the first type of point cluster can be filtered. In step S1222, if the existence status indicates that the second type of point cluster is not empty, the first type of point cluster is cleared and the second type of point cluster is filtered.

[0100] Specifically, if the second type of point cluster does not exist, then the point cluster C i There are no points outside the image. If there is one intersection in the first type of point cluster, the intersection is directly retained. If there are multiple intersections in the first type of point cluster, the intersection whose coordinates are on the edge of the third document is retained. If none of the multiple intersections in the first type of point cluster are on the edge of the third document, the intersection closest to the center of the document image to be processed is retained. The intersections on the edge of the third document and the intersections not on the edge of the third document are as follows. Figure 11 As shown in Figures C and D in the figure.

[0101] If there are intersections in the second type of point cluster, indicating that the document corner is missing due to incomplete capture, the first type of point cluster is cleared and the second type of point cluster is filtered. The specific filtering method includes: determining the number of intersections contained in the second type of point cluster; if there is a single intersection in the second type of point cluster, retaining it directly; if there are multiple intersections in the second type of point cluster, retaining the intersection closest to the center of the document image to be processed.

[0102] Furthermore, the four vertices of the document in the document image to be processed are determined based on the centers of the filtered multi-class point clusters. Specifically, after obtaining the four filtered point clusters, the center of each class of point clusters is calculated. The coordinates of the centers of the four class of point clusters are the coordinates of the four vertices of the document image to be processed. The coordinates of the four vertices can be expressed as P = {(x i ,y i ),1≤i≤4}, thus completing the vertex positioning of the document in the document image to be processed, such as Figure 11 As shown in Figures E and F in the figure.

[0103] Continue to refer Figure 2 As shown in , in step S240 , the document image to be processed is filled and perspective transformed in combination with the four vertices to obtain a document correction result.

[0104] In an embodiment of the present disclosure, after determining the four vertices of the document, the document image to be processed can be subjected to a perspective transformation in combination with the four vertices to obtain a document correction result corresponding to the document image to be processed. Perspective transformation is used to eliminate perspective distortion (distortion) and correct the image to a front view. For example, an object is actually a square, but it appears to be a trapezoid or other shape. The perspective transformation can make the two-dimensional shape in the three-dimensional space show its own visual effect, so that the seen two-dimensional shape is converted into a standard shape. For example, a trapezoidal image can be converted into a standard rectangle through perspective transformation.

[0105] Before performing perspective transformation, additional judgment and processing can be performed first. Specifically, it can be judged whether the quadrilateral formed by the four vertices is reasonable. The judgment conditions are: whether the area ratio of the quadrilateral formed by the four vertices to the entire image is greater than 10%, and whether the four internal angles of the quadrilateral are within the range of [50°, 130°]. If the quadrilateral meets the above two conditions, the subsequent perspective transformation will continue; otherwise, the subsequent steps will not be performed and the original image will be returned directly. The flowchart of perspective transformation can be referred to Figure 13 As shown in , it mainly includes the following steps:

[0106] In step S1301, if it is determined to continue the perspective transformation, the padding width required for the document image to be processed in multiple directions is calculated based on the coordinates of the four vertices to perform the padding operation. The process of calculating the padding width includes the following steps:

[0107] The first step is to define the padding widths required for the document image to be processed in the four directions of top, bottom, left, and right and initialize them to 0, i.e. n_top = 0, n_bottom = 0, n_left = 0, n_right = 0. The second step is to loop through the four vertex coordinates (x i ,y i ), update the above four padding widths, i.e. padding values. The specific process includes the following cases:

[0108] The first one is if x i <0, then n_left=(0-xi)>n_left? (0-xi):n_left, that is, if (0-xi)>n_left holds, then n_left=(0-xi):, otherwise n_left=n_left.

[0109] The second type, if x i≥img_w, then n_right=(xi-img_w+1)>n_right? (xi-img_w+1):n_right, img_w represents the width of the image, that is, xi-img_w+1>n_right holds, n_right=xi-img_w+1. Otherwise, n_right=n_right.

[0110] The third type, if y i <0, then n_top=(0-yi)>n_top? (0-yi):n_top, that is, 0-yi>n_top holds, then n_top=0–yi, otherwise n_top=n_top.

[0111] The fourth type, if y i ≥img_h, then n_bottom = (yi - img_h + 1) > n_bottom? (yi - img_h + 1): n_bottom, img_h represents the height of the image. That is, yi - img_h + 1 > n_bottom?, then n_bottom = yi - img_h + 1, otherwise n_bottom = n_bottom.

[0112] Perform padding operation based on the updated padding width to obtain the padded document image to be processed. Perform padding operation based on the updated 4 padding widths to obtain the padded document image to be processed. Figure 14 As shown in Figure A, since the document image to be processed does not have the "corner missing" problem, that is, the four vertices of the document are all within the image range, then the four padding width values ​​are all 0, that is, after the padding operation, the document image to be processed remains unchanged, and it can be considered that no padding operation is required. The result of the padding operation is shown in Figure 14 As shown in Figure B in the figure, the perspective transformation result is as follows Figure 14 As shown in Figure C. Figure 15 The document image to be processed shown in the figure has the problem of "missing corners", which causes the lower left and lower right document vertices to be outside the image. Therefore, padding needs to be performed on the lower and right sides of the image. The result of the padding operation is as follows: Figure 15 As shown in Figure B in the figure, the perspective transformation result is as follows Figure 15 As shown in Figure C.

[0113] If a fill operation is performed on the document image to be processed, for example, a fill operation is performed in a preset direction. The preset direction can be the left or the top, and the corresponding fill width is not 0. Therefore, the coordinates of the four vertices need to be updated according to the updated fill width, which can be expressed as x′ i =x i +n_left,y′ i=y i +n_top. Then, the four vertices of the document in the updated document image to be processed are P′={(x′ i ,y′ i ),1≤i≤4}.

[0114] In step S1302, four target vertices of the document in the document image to be processed within the target plane are determined according to the updated coordinates of the four vertices.

[0115] In this step, the four updated vertices P′ are sorted clockwise, ensuring that the first vertex is closest to the image origin. Since the document region is rectangular within the target plane after perspective transformation, only the top-left and bottom-right vertices of the document image to be processed within the target plane need to be calculated. Therefore, the target vertices refer to the top-left and bottom-right vertices of the document. The target plane is the plane where the original document image to be processed lies after perspective transformation.

[0116] In order to complete the clipping operation during the perspective transformation, the coordinates of the upper left vertex can be set to (0,0), based on which the lower right vertex The calculation method is as follows: 1. Calculate the horizontal distance between the first and second document vertices w1 = x′2-x′1, calculate the horizontal distance between the third and fourth document vertices w2 = x′3-x′4, and take their average As the x-axis coordinate of the lower right vertex; 2. Similarly, calculate the vertical distance between the first and fourth document vertices h1 = y′4 - y′1, calculate the vertical distance between the second and third document vertices h2 = y′3 - y′2, and take their average As the y-axis coordinate of the lower right vertex. In this way, the four target vertices of the document area in the target plane can be obtained.

[0117] In step S1303, a perspective transformation matrix is ​​calculated based on the updated four vertices and the four target vertices, and a perspective transformation is performed on the filled document image to be processed based on the perspective transformation matrix to perform correction.

[0118] The perspective transformation matrix is ​​used to describe the mapping relationship between the four updated vertices and the four target vertices. The perspective transformation matrix M is calculated using the four updated vertices P' and the four target vertices Q.

[0119] When calculating the perspective transformation matrix, assume that the four vertices are V1, V2, V3, and V4, and their coordinates are (x1, y1), (x2, y2), (x3, y3), and (x4, y4), respectively. V1 is the vertex closest to the document image (0, 0), and these four vertices are arranged strictly clockwise. Assume that the four target vertices are V1′, V′2, V′3, and V′4, and their coordinates can be expressed as (x′1, y′1), (x′2, y′2), (x′3, y′3), and (x′4, y′4).

[0120] Taking V1 and V1′ as examples, according to the perspective transformation principle, assuming that their corresponding coordinates in three-dimensional space are (X1, Y1, Z1), the process of perspective transformation of V1 into three-dimensional space can be expressed as formula (4):

[0121]

[0122] in, is the perspective transformation matrix.

[0123] Project (X1, Y1, Z1) onto the target plane to get V1′, then we have It can be expressed as shown in formula (5):

[0124]

[0125] Another 33 =1, expand the above formula to get formula (6):

[0126]

[0127] It can be seen that one set of original vertices and target vertices can determine two equations. Similarly, using the other three sets of vertices can get another six equations. Therefore, using 4 sets of original vertices and target vertices can get 8 equations, and using these 8 equations we can solve a 11 ~a 32 These 8 unknowns are used to solve the perspective transformation matrix M.

[0128] Then, the filled document image to be processed is perspective transformed according to the perspective transformation matrix M, thereby completing the document correction. Figure 16 As shown in FIG, document edge detection is performed on the document image to be processed in FIGA to obtain the document edge shown in FIGB; straight line fitting is performed on the document edge to obtain a straight line set as shown in FIGC; vertex positioning is performed on the straight line set to obtain the four vertices of the document in the document image to be processed as shown in FIGD; perspective transformation is performed based on the four vertices to obtain the document correction result, as shown in FIGE.

[0129] The technical solution in the disclosed embodiments is that the document edge detection model DocEdgeNet based on a convolutional neural network is more robust than traditional edge detection methods. In order to train a better model, edge-aware loss and semantic segmentation loss are designed. The former can solve the problem of imbalance between the categories of "edge pixels" and "non-edge pixels", and the latter can effectively suppress the interference of erroneous edges in the background or document area, thereby improving the accuracy of edge detection. In addition, DocEdgeNet is extremely lightweight, with a model weight of only 360K and FLOPs of only 0.085M. Coupled with the high efficiency of the subsequent processing process, it can run in real time on mobile phones. The document boundary straight line fitting and document positioning are highly reliable; by adding a padding operation, it can effectively address the problem of missing corners, so that the corrected text lines remain horizontal, improving accuracy. It can be applied to various types of document images, improving versatility and convenience, and improving image correction effects. It can perform perspective correction on various types of document images, facilitating reading or archiving, and improving versatility. In addition, it can also be applied to the preprocessing process of OCR (Optical character recognition), which can significantly improve the accuracy of text recognition, and thus be applied in scenarios such as document scanning, business card recognition, and photo translation.

[0130] In the embodiment of the present disclosure, a document image processing device is also provided. Figure 17 As shown in , the document image processing device 1700 specifically includes the following modules:

[0131] The edge acquisition module 1701 is used to acquire a document image to be processed and perform document edge detection on the document image to be processed to obtain a document edge;

[0132] A line set determination module 1702 is configured to perform line fitting on the document edge to determine a line set;

[0133] A document vertex positioning module 1703 is configured to determine four vertices of a document in the original document image to be processed based on the set of straight lines;

[0134] The image correction module 1704 performs a filling operation and a perspective transformation on the document image to be processed based on the four vertices to obtain a document correction result.

[0135] In an exemplary embodiment of the present disclosure, the edge acquisition module includes: a first acquisition module, used to perform document edge detection on the document image to be processed through an edge detection model to obtain a first document edge; a second acquisition module, used to binarize the first document edge to obtain a second document edge; a third acquisition module, used to filter the second document edge to obtain a third document edge; and a fourth acquisition module, used to refine the third document edge to obtain a fourth document edge as the document edge.

[0136] In an exemplary embodiment of the present disclosure, the first acquisition module includes: a feature map acquisition module, which is used to obtain feature maps of different resolutions from the encoder and the decoder respectively; a convolution module, which is used to perform dimensionality reduction convolution and upsampling on the feature maps to obtain upsampled feature maps so that the feature maps have the same width and height; a fusion module, which is used to perform feature fusion on the upsampled feature maps to obtain fused features, and perform document edge prediction based on the fused features to obtain a first document edge.

[0137] In an exemplary embodiment of the present disclosure, the device also includes: a model training module, which is used to use training data to determine a loss function based on edge perception loss and semantic segmentation loss, and train the deep learning model with the minimum loss function as the training goal to obtain the edge detection model.

[0138] In an exemplary embodiment of the present disclosure, the straight line set determination module includes: a first set determination module, used to perform straight line detection on the edge of the document to determine a first straight line set; a second set determination module, used to cluster the straight lines in the first straight line set to determine a straight line cluster for line merging, and obtain a second straight line set based on the center of the straight line cluster; a third set determination module, used to filter the straight lines in the second straight line set according to a filtering condition, and obtain a third straight line set as the straight line set.

[0139] In an exemplary embodiment of the present disclosure, the second set determination module includes: a sorting module, which is used to sort the straight lines in the first straight line set in ascending order according to the distance from the coordinate origin of the document image to be processed to the straight line; a first straight line cluster determination module, which is used to determine the first type of straight line cluster based on the sorted first straight line; a straight line cluster determination module, which is used to obtain the distance difference and angle difference between the next straight line and the center of the last type of straight line cluster, determine the next straight line as a target type straight line cluster based on the distance difference and the angle difference, and obtain a straight line cluster based on the first type of straight line cluster and the target type straight line cluster; a merging module, which is used to calculate the center of each type of straight line cluster in the straight line cluster, and determine the set formed by the center of each type of straight line cluster as the straight line merging result to determine the second straight line set.

[0140] In an exemplary embodiment of the present disclosure, the document vertex positioning module includes: a candidate intersection set determination module, which is used to calculate the angle between every two straight lines in the straight line set to determine the intersection of every two straight lines, and form a candidate intersection set based on the intersection; an intersection aggregation module, which is used to aggregate all intersections in the candidate intersection set into multiple point clusters; and a vertex determination module, which is used to filter the multiple point clusters and determine the four vertices of the document in the document image to be processed based on the centers of the filtered multiple point clusters.

[0141] In an exemplary embodiment of the present disclosure, the vertex determination module includes: an intersection division module, used to divide the intersection points in the multiple types of point clusters into first type point clusters and second type point clusters according to the range of the intersection points and the document image to be processed; an intersection filtering module, used to filter the first type point cluster or the second type point cluster according to the existence status of the second type point cluster.

[0142] In an exemplary embodiment of the present disclosure, the intersection filtering module includes: a retention module, which is used to directly retain the intersection if the second type of point cluster does not exist and there is one intersection in the first type of point cluster; a first filtering module, which is used to retain the intersection whose coordinates are on the edge of the third document if there are multiple intersections in the first type of point cluster; and a second filtering module, which is used to retain the intersection closest to the center of the document image to be processed if none of the multiple intersections in the first type of point cluster are on the edge of the third document.

[0143] In an exemplary embodiment of the present disclosure, the intersection filtering module includes: a quantity determination module, which is used to clear the first type of point cluster and determine the number of intersections contained in the second type of point cluster if there is an intersection in the second type of point cluster; a third filtering module, which is used to directly retain the intersection if there is one intersection in the second type of point cluster; and a fourth filtering module, which is used to retain the intersection closest to the center of the document image to be processed if there are multiple intersections in the second type of point cluster.

[0144] In an exemplary embodiment of the present disclosure, the image correction module includes: a filling module, which is used to perform a filling operation according to the filling width of the document image to be processed to update the coordinates of the four vertices if it is determined that the quadrilateral formed by the four vertices is reasonable; a target vertex determination module, which is used to determine the four target vertices of the document image to be processed in the target plane according to the coordinates of the updated four vertices; a matrix calculation, which is used to calculate the perspective transformation matrix according to the updated four vertices and the four target vertices, and perform perspective transformation on the filled document image to be processed according to the perspective transformation matrix to perform correction to obtain a document correction result.

[0145] It should be noted that the specific details of each module in the above-mentioned document image processing device have been described in detail in the corresponding image processing method, and therefore will not be repeated here.

[0146] Figure 18 Schematic diagram of an electronic device suitable for implementing an exemplary embodiment of the present disclosure is shown. The terminal of the present disclosure can be configured as follows Figure 18 The form of the electronic device shown, however, needs to be explained. Figure 18 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0147] The electronic device of the present disclosure includes at least a processor and a memory, where the memory is used to store one or more programs. When the one or more programs are executed by the processor, the processor can implement the method of the exemplary embodiment of the present disclosure.

[0148] Specifically, such as Figure 18 As shown, the electronic device 1800 may include: a processor 1810, an internal memory 1821, an external memory interface 1822, a Universal Serial Bus (USB) interface 1830, a charging management module 1840, a power management module 1841, a battery 1842, an antenna 1, an antenna 2, a mobile communication module 1850, a wireless communication module 1860, an audio module 1870, a speaker 1871, a receiver 1872, a microphone 1873, an earphone interface 1874, a sensor module 1880, a display screen 1890, a camera module 1891, an indicator 1892, a motor 1893, a button 1894 and a subscriber identification module (Subscriber Identification Module, SIM) card interface 1895, etc. The sensor module 1880 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor and a bone conduction sensor, etc.

[0149] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 1800. In other embodiments of the present application, the electronic device 1800 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0150] The processor 1810 may include one or more processing units, for example: the processor 1810 may include an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a video codec, a digital signal processor, a baseband processor and / or a neural network processor (Neural-etwork Processing Unit, NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. In addition, a memory can be provided in the processor 1810 for storing instructions and data. The image processing method in this exemplary embodiment can be executed by an application processor, a graphics processor or an image signal processor. When the method involves processing related to a neural network, it can be executed by an NPU.

[0151] Internal memory 1821 can be used to store computer executable program code, which includes instructions. Internal memory 1821 can include a program storage area and a data storage area. External memory interface 1822 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of electronic device 1800.

[0152] The communication functions of mobile terminal 1800 are implemented through a mobile communication module, antenna 1, a wireless communication module, antenna 2, a modem processor, and a baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module can provide 2G, 3G, 4G, and 5G mobile communication solutions for mobile terminal 1800. The wireless communication module can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 200.

[0153] The display module is used to implement display functions, such as displaying user interfaces, images, and videos. The camera module is used to implement shooting functions, such as capturing images and videos. The audio module is used to implement audio functions, such as playing audio and capturing voice. The power module is used to implement power management functions, such as charging the battery, powering the device, and monitoring the battery status.

[0154] The present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.

[0155] Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device.

[0156] Computer-readable storage media can transmit, propagate, or transfer programs for use by or in conjunction with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0157] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments.

[0158] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0159] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0160] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0161] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims. It should be understood that the present disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and that various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A document image processing method, characterized in that: include: Acquire a document image to be processed, and perform document edge detection on the document image to be processed to obtain a document edge; Performing straight line fitting on the document edge to determine a straight line set includes: performing straight line detection on the document edge to determine a first straight line set; clustering the straight lines in the first straight line set to determine a straight line cluster for line merging, and obtaining a second straight line set based on the center of the straight line cluster; filtering the straight lines in the second straight line set according to a filtering condition to obtain a third straight line set as the straight line set; Determining four vertices of the document in the document image to be processed based on the straight line set includes: calculating the angle between each two straight lines in the straight line set to determine the intersection of each two straight lines, and forming a candidate intersection set based on the intersection points; aggregating all intersection points in the candidate intersection set into multiple point clusters; filtering the multiple point clusters, and determining four vertices of the document in the document image to be processed based on the centers of the filtered multiple point clusters; Performing a filling operation and a perspective transformation on the document image to be processed in combination with the four vertices to obtain a document correction result; The filtering of the multi-class point clusters includes: Dividing the intersection points in the plurality of point clusters into a first type of point cluster and a second type of point cluster according to the range between the intersection points and the document image to be processed; the first type of point cluster is a point cluster located within the image, and the second type of point cluster is a point cluster located outside the image; The first type of point cluster or the second type of point cluster is filtered according to the existence status of the second type of point cluster, where the existence status is used to indicate whether the second type of point cluster is empty.

2. The document image processing method according to claim 1, wherein: The performing document edge detection on the document image to be processed to obtain the document edge includes: Performing document edge detection on the document image to be processed by using an edge detection model to obtain a first document edge; performing binarization processing on the first document edge to obtain a second document edge; Filtering the second document edge to obtain a third document edge; The third document edge is thinned to obtain a fourth document edge as the document edge.

3. The document image processing method according to claim 2, wherein: The performing document edge detection on the document image to be processed by using the edge detection model to obtain a first document edge includes: Obtain feature maps of different resolutions from the encoder and decoder respectively; Performing dimensionality reduction convolution and upsampling on the feature map to obtain an upsampled feature map so that the feature maps have the same width and height; The upsampled feature map is subjected to feature fusion to obtain a fusion feature, and a document edge is predicted based on the fusion feature to obtain a first document edge.

4. The document image processing method according to claim 2 or 3, characterized in that: The method further comprises: Using the training data, a loss function is determined according to the edge perception loss and the semantic segmentation loss, and the deep learning model is trained with the minimum loss function as the training goal to obtain the edge detection model.

5. The document image processing method according to claim 1, wherein: The clustering of the lines in the first line set to determine a line cluster for line merging, and obtaining a second line set according to the center of the line cluster, includes: Sorting the straight lines in the first straight line set in ascending order of the distance from the coordinate origin of the document image to be processed to the straight line; Determine the first type of straight line cluster based on the first straight line after sorting; Obtaining a distance difference and an angle difference between the next straight line and the center of the last straight line cluster, determining the next straight line as a target straight line cluster based on the distance difference and the angle difference, and obtaining a straight line cluster based on the first straight line cluster and the target straight line cluster; the target straight line cluster is the newly created straight line cluster or the last straight line cluster; The center of each type of line cluster in the line clusters is calculated, and a set consisting of the centers of each type of line cluster is determined as a line merging result to determine a second line set.

6. The document image processing method according to claim 1, wherein: The filtering of the first type of point clusters or the second type of point clusters according to the existence status of the second type of point clusters includes: If the second type of point cluster does not exist and the first type of point cluster has an intersection point, directly retain the intersection point; If there are multiple intersection points in the first type of point cluster, retain the intersection point whose coordinates are on the edge of the third document; If the plurality of intersection points in the first type of point cluster are not located on the edge of the third document, the intersection point closest to the center of the document image to be processed is retained.

7. The document image processing method according to claim 1, wherein: The filtering of the first type of point cluster or the second type of point cluster according to the existence status of the second type of point cluster includes: If there are intersections in the second type of point cluster, clear the first type of point cluster and determine the number of intersections contained in the second type of point cluster; If there is an intersection point in the second type of point cluster, directly retain the intersection point; If there are multiple intersection points in the second type of point cluster, the intersection point closest to the center of the document image to be processed is retained.

8. The document image processing method according to claim 1, wherein: The step of performing a filling operation and a perspective transformation on the document image to be processed by combining the four vertices to obtain a document correction result includes: If it is determined that the quadrilateral formed by the four vertices is reasonable, a filling operation is performed according to the filling width of the document image to be processed to update the coordinates of the four vertices; Determining four target vertices of the document in the document image to be processed within the target plane according to the updated coordinates of the four vertices; A perspective transformation matrix is ​​calculated based on the updated four vertices and the four target vertices, and a perspective transformation is performed on the filled document image to be processed based on the perspective transformation matrix to perform correction to obtain a document correction result.

9. A document image processing device, characterized in that: include: An edge acquisition module is used to acquire a document image to be processed and perform document edge detection on the document image to be processed to obtain a document edge; a line set determination module, configured to perform line fitting on the document edge to determine a line set, including: performing line detection on the document edge to determine a first line set; clustering the lines in the first line set to determine line clusters for line merging, and obtaining a second line set based on the centers of the line clusters; and filtering the lines in the second line set according to a filtering condition to obtain a third line set as the line set; A document vertex positioning module is configured to determine four vertices of a document in the original document image to be processed based on the set of straight lines, comprising: calculating the angle between each two straight lines in the set of straight lines to determine the intersection of each two straight lines, and forming a candidate intersection set based on the intersections; aggregating all intersections in the candidate intersection set into multiple point clusters; filtering the multiple point clusters, and determining four vertices of the document in the document image to be processed based on the centers of the filtered multiple point clusters; An image correction module, combining the four vertices to perform a filling operation and a perspective transformation on the document image to be processed to obtain a document correction result; The filtering of the multi-class point clusters includes: Dividing the intersection points in the plurality of point clusters into a first type of point cluster and a second type of point cluster according to the range between the intersection points and the document image to be processed; the first type of point cluster is a point cluster located within the image, and the second type of point cluster is a point cluster located outside the image; The first type of point cluster or the second type of point cluster is filtered according to the existence status of the second type of point cluster, where the existence status is used to indicate whether the second type of point cluster is empty.

10. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the document image processing method according to any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the document image processing method according to any one of claims 1 to 8 is implemented.

Images