A Method and System for Mobile Document Edge Detection and Correction Based on Semantic Segmentation
By using a semantic segmentation-based method, the edges of mobile documents are detected and corrected, solving the problem of unstable edge detection under complex conditions and achieving accurate restoration of the document's physical aspect ratio and perspective correction.
Patent Information
- Application Number
- CN202611567155.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-10
- Publication Date
- 2026-11-17
Smart Images

Figure CN123118348A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of mobile terminal image processing technology, specifically to a method and system for mobile terminal document edge detection and correction based on semantic segmentation. Background Technology
[0002] With the rapid development of mobile internet and smart terminal technologies, the camera performance and computing power of smartphones have continued to improve, making it possible to digitize physical media such as paper documents, books, certificates, and cards using mobile phones. Traditional scanning methods typically rely on dedicated flatbed scanners or portable scanning devices. While these devices can achieve high image quality, they have limitations such as large size, inconvenience in carrying, cumbersome operation procedures, and inability to scan anytime, anywhere. In contrast, the high-pixel cameras built into mobile phones, combined with image processing algorithms, can quickly complete the image acquisition and preliminary processing of paper documents, book pages, ID cards, bank cards, and various other cards. Through edge detection, perspective correction, noise reduction and enhancement, and OCR text recognition technologies, documents can be digitized for storage, retrieval, and sharing.
[0003] While mobile phone scanning offers significant advantages in portability and immediacy, in actual shooting, the camera's optical axis is often difficult to keep perfectly perpendicular to the document's plane. As a result, a planar rectangular document, after being projected through a pinhole camera, often no longer appears as a rectangle in the image, but rather as an arbitrary convex quadrilateral. Furthermore, the pixel lengths of the top, bottom, left, and right edges of the same document will change under different pitch angles, yaw angles, and shooting distances, posing difficulties for subsequent geometric correction and layout restoration.
[0004] To address the aforementioned issues, existing mobile document scanning methods typically first locate candidate quadrilaterals using methods such as grayscale conversion, binarization, edge detection, or contour extraction, and then map these quadrilaterals into rectangles using four-point homography. For the width and height of the output rectangle, a common approach is to determine them based on the longest side, adjacent sides, or the average pixel length of opposite sides. However, this method has the following drawbacks and limitations:
[0005] (1) Unstable edges in weak contrast scenes: Traditional methods such as grayscale, threshold and contour are sensitive to paper color, background texture, shadow, reflection and occlusion, and are prone to obtaining non-document contours or incomplete edges.
[0006] (2) Four-point flattening is not equal to proportional restoration: Directly performing homography transformation can only ensure that the quadrilateral is mapped to a rectangle. The output width and height are still often determined by the pixel side lengths that are compressed by perspective, and cannot reflect the physical aspect ratio of the original paper.
[0007] (3) Lack of constraints related to camera physical parameters: The transformation matrix is determined by only four two-dimensional points, which cannot distinguish the difference between the physical width and height of the original and the projection shortening caused by the camera posture.
[0008] (4) Insufficient adaptation of output direction and resolution: Traditional solutions require manual selection of horizontal and vertical directions, or only set the output size according to the original pixel side length, making it difficult to obtain consistent results in batch scanning.
[0009] (5) Complex pose calculation is not suitable for direct use on mobile devices: High degree of freedom pose calculation usually requires accurate calibration, known physical dimensions or stable corner input; in the real-time processing environment of consumer mobile devices, corner errors are easily amplified. Summary of the Invention
[0010] To address this, this application provides a method and system for edge detection and correction of mobile documents based on semantic segmentation, in order to solve the problems of unstable edge detection in existing mobile document scanning methods under conditions of weak contrast, complex background, shadow, reflection and partial occlusion, and the inability to restore the true physical aspect ratio of the paper by simply flattening the document according to the pixel side length.
[0011] To achieve the above objectives, this application provides the following technical solution:
[0012] Firstly, a method for edge detection and correction of mobile documents based on semantic segmentation includes:
[0013] Step 1: Obtain the image of the document to be processed and the mobile camera parameters, and calculate the pixel focal length based on the camera parameters; the camera parameters include the principal point;
[0014] Step 2: Perform data preprocessing on the document image to be processed, and input the preprocessed document image to be processed into a pre-trained lightweight semantic segmentation model to obtain the document foreground mask; the lightweight semantic segmentation model is configured to run locally on the mobile device;
[0015] Step 3: Perform contour detection and convex hull processing on the document foreground mask to obtain multiple candidate contours, calculate the comprehensive score of each candidate contour, and select the candidate contour with the highest comprehensive score.
[0016] Step 4: Perform multi-threshold polygon approximation on the candidate contour with the highest comprehensive score to obtain a quadrilateral. Restore the four points of the quadrilateral from the model coordinate system to the original image coordinate system, and sort them by polar angle based on the geometric center of the four points. Then rotate the upper left corner to the first position to obtain an ordered four corner points arranged in the order of upper left, upper right, lower right, and lower left.
[0017] Step 5: Perform validity verification on the ordered four corner points. If the verification passes, expand the points by a finite number of pixels and constrain the expanded points within the range of the input image.
[0018] Step 6: Based on the points constrained within the input image range, calculate the Euclidean lengths of the top, bottom, left, and right sides of the quadrilateral, and calculate the original pixel aspect ratio based on the average lengths of the top and bottom sides and the average lengths of the left and right sides.
[0019] Step 7: Find the intersection of the upper and lower lines to obtain the horizontal vanishing point, and find the intersection of the left and right lines to obtain the vertical vanishing point. Calculate the distances from the horizontal and vertical vanishing points to the principal point, and combine the pixel focal length to calculate the perspective shortening coefficients in the width and height directions.
[0020] Step 8: Calculate the restored physical aspect ratio based on the original pixel aspect ratio, the perspective shortening factor in the width direction, and the perspective shortening factor in the height direction, and limit the restored physical aspect ratio within a preset range; determine the output width and output height of the target rectangle based on the average length of the top and bottom sides and the average length of the left and right sides of the quadrilateral, using the reference pixel count; construct a homography matrix using the ordered four corner points as the source points and the vertices of the target rectangle as the target points, and perform perspective transformation to obtain the corrected rectangular document image.
[0021] Preferably, in step 1, the camera parameters further include: physical focal length, sensor physical width, and image pixel width, wherein the formula for calculating the pixel focal length is:
[0022] f = F_mm x W_img / S_w_mm
[0023] Where f represents the pixel focal length, F_mm represents the physical focal length, W_img represents the image pixel width, and S_w_mm represents the sensor physical width.
[0024] Preferably, in step 2, the data preprocessing of the document image to be processed specifically includes: first, scaling the document image to be processed according to a preset side length while maintaining the ratio, and padding the sides to the input size of the lightweight semantic segmentation model; then, converting the image to RGB format and normalizing it according to the mean and standard deviation during training.
[0025] Preferably, in step 2, the lightweight semantic segmentation model adopts a binary classification model based on PP-MobileSeg; the binary classification model based on PP-MobileSeg is converted by ONNX and run locally on the mobile device in MNN format.
[0026] Preferably, in step 3, when calculating the comprehensive score for each candidate contour, the calculation formula is as follows:
[0027] Score = w1 x RectScore + w2 x RatioScore + w3 x AreaScore + w4 xCentralityScore
[0028] Wherein, Score represents the overall score, RectScore represents the ratio of the area of the candidate contour to the area of its minimum bounding rectangle, RatioScore represents the closeness of the aspect ratio of the candidate contour to the preset target interval, AreaScore represents the proportion of the candidate contour area to the effective image area, and CentralityScore represents the normalized score of the distance between the center of the candidate contour and the center of the image. w1, w2, w3 and w4 represent the weights, respectively.
[0029] Preferably, in step 4, when performing multi-threshold polygon approximation on the candidate contour with the highest comprehensive score, if the number of vertices obtained from the approximation is less than or greater than four, then the quadrilateral is obtained by backtracking using the minimum bounding rectangle.
[0030] Preferably, in step 5, the validity verification of the ordered four corner points specifically includes: verifying the number, convexity, non-self-intersection, area ratio, and side length of the ordered four corner points. If the area is too small, too large, the side length is zero, or four valid corner points are not obtained, the original image is returned, prompting for re-detection, or a rollback process without proportional compensation is entered.
[0031] Preferably, in step 7, when the top and bottom edges are approximately parallel and the vanishing point in the horizontal direction cannot be stably calculated, the perspective shortening factor in the width direction is set to one; when the left and right edges are approximately parallel and the vanishing point in the vertical direction cannot be stably calculated, the perspective shortening factor in the height direction is set to one.
[0032] Preferably, in step 7, the formula for calculating the perspective shortening factor in the width direction is:
[0033] k_w = d_h / sqrt(d_h^2 + f^2)
[0034] Where k_w represents the perspective shortening factor in the width direction, d_h represents the distance from the vanishing point to the principal point in the horizontal direction, and f represents the pixel focal length;
[0035] The formula for calculating the perspective shortening factor in the height direction is:
[0036] k_h = d_v / sqrt(d_v^2 + f^2)
[0037] Where k_h represents the perspective shortening factor in the height direction, and d_v represents the distance from the vanishing point in the vertical direction to the principal point.
[0038] Secondly, a mobile document edge detection and correction system based on semantic segmentation includes:
[0039] An image acquisition module is used to acquire the document image to be processed and the mobile camera parameters, and to calculate the pixel focal length based on the camera parameters; the camera parameters include the principal point;
[0040] The document segmentation module is used to preprocess the document image to be processed, and input the preprocessed document image to be processed into a pre-trained lightweight semantic segmentation model to obtain a document foreground mask; the lightweight semantic segmentation model is configured to run locally on the mobile device;
[0041] The four-corner point determination module is used to perform contour detection and convex hull processing on the document foreground mask to obtain multiple candidate contours, calculate the comprehensive score of each candidate contour, and select the candidate contour with the highest comprehensive score.
[0042] And perform multi-threshold polygon approximation on the candidate contour with the highest comprehensive score to obtain a quadrilateral. Restore the four points of the quadrilateral from the model coordinate system to the original image coordinate system, and sort them by polar angle based on the geometric center of the four points. Then rotate the upper left corner to the first position to obtain the ordered four corner points arranged in the order of upper left, upper right, lower right, and lower left.
[0043] The scaling recovery module is used to verify the validity of the ordered four corner points. After the verification is passed, the points are expanded outward by a finite number of pixels, and the expanded points are constrained within the range of the input image.
[0044] And based on the points constrained within the input image range, calculate the Euclidean lengths of the top, bottom, left, and right sides of the quadrilateral, and calculate the original pixel aspect ratio based on the average lengths of the top and bottom sides and the average lengths of the left and right sides.
[0045] The perspective transformation module is used to find the intersection of the upper and lower lines to obtain the horizontal vanishing point, and to find the intersection of the left and right lines to obtain the vertical vanishing point. It also calculates the distances from the horizontal and vertical vanishing points to the principal point, and calculates the perspective shortening coefficients in the width and height directions in combination with the pixel focal length.
[0046] The system calculates the restored physical aspect ratio based on the original pixel aspect ratio, the perspective shortening factor in the width direction, and the perspective shortening factor in the height direction, and limits the restored physical aspect ratio within a preset range; it determines the output width and output height of the target rectangle based on the average length of the top and bottom sides and the average length of the left and right sides of the quadrilateral, using the reference pixel count; it constructs a homography matrix using the ordered four corner points as source points and the vertices of the target rectangle as target points, and performs perspective transformation to obtain the corrected rectangular document image.
[0047] Compared with the prior art, this application has at least the following beneficial effects:
[0048] 1. This application provides a method for mobile document edge detection and correction based on semantic segmentation, including: acquiring the document image to be processed and mobile camera parameters, and calculating the pixel focal length; inputting the document image to be processed into a lightweight semantic segmentation model to obtain a document foreground mask; performing contour detection and convex hull processing on the document foreground mask to obtain multiple candidate contours, calculating the comprehensive score of each candidate contour, selecting the candidate contour with the highest comprehensive score for multi-threshold polygon approximation to obtain a quadrilateral; restoring the four points of the quadrilateral to the original image coordinate system, and sorting them by polar angle based on the geometric center of the four points, and then rotating the upper left corner. The method first obtains ordered four corner points. These points are then validated, and if valid, expanded by a finite number of pixels, confining the expanded points within the input image range. Based on these points, the Euclidean lengths of the top, bottom, left, and right sides of the quadrilateral are calculated, along with the original pixel aspect ratio. The horizontal and vertical vanishing points are calculated, and their distances to the principal point are calculated. Perspective shortening coefficients in the width and height directions are calculated using the pixel focal length. The restored physical aspect ratio is calculated, a homography matrix is constructed, and perspective transformation is performed to obtain the corrected rectangular document image. This method provides stable detection of document edges and output of ordered four corner points even under conditions of weak contrast, complex backgrounds, shadows, reflections, and partial occlusion. Furthermore, by compensating for vanishing points and focal length, the true physical aspect ratio of the paper is restored, resulting in more accurate perspective correction.
[0049] 2. Lightweight networks like PP-MobileSeg can run locally on mobile devices such as Android / iOS after being converted by ONNX / MNN; contour filtering and corner recovery use only limited geometric operations, avoiding uploading the original image to the server. Attached Figure Description
[0050] To more intuitively illustrate the prior art and this application, exemplary drawings are provided below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary drawings, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).
[0051] Figure 1 A flowchart of a mobile document edge detection and correction method based on semantic segmentation provided in Embodiment 1 of this application;
[0052] Figure 2 A schematic diagram of the structure of a mobile document edge detection and correction method based on semantic segmentation provided in Embodiment 1 of this application;
[0053] Figure 3 This is a schematic diagram of the structure from the document foreground mask to the ordered four corner points provided in Embodiment 1 of this application;
[0054] Figure 4 A schematic diagram illustrating the geometric relationship between the four corner points, vanishing point, and perspective scale restoration of a document, provided as an example of Embodiment 1 of this application;
[0055] Figure 5 This is a schematic diagram illustrating the document edge detection operation effect provided in Example 1 of this application;
[0056] Figure 6 This is a schematic diagram illustrating the perspective transformation effect provided in Example 1 of this application;
[0057] Figure 7 This is a schematic diagram of a mobile document edge detection and correction system based on semantic segmentation, provided in Embodiment 2 of this application. Detailed Implementation
[0058] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0059] In the description of this application: unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," "third," etc., in this application are intended to distinguish the objects referred to and do not have any special meaning in terms of technical connotation (e.g., they should not be construed as an emphasis on importance or order). Expressions such as "including," "comprising," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).
[0060] The terms used in this application, such as "upper," "lower," "left," "right," and "middle," are generally used to indicate the general relative positional relationship for the purpose of intuitive understanding by referring to the accompanying drawings, and are not absolute limitations on the positional relationship in the actual product.
[0061] Example 1
[0062] Please see Figure 1 and Figure 2 This embodiment provides a method for mobile document edge detection and correction based on semantic segmentation, including:
[0063] S1, Obtain the document image to be processed and camera parameters: Obtain the document image to be processed and the mobile camera parameters, and calculate the pixel focal length based on the camera parameters; the camera parameters include the principal point;
[0064] Specifically, this step first acquires the document image I to be processed, with a width of W_img and a height of H_img. For a camera-captured scene, camera parameters are obtained, including the physical focal length F_mm, the sensor physical width S_w_mm, the image pixel width W_img, and the principal point. Then, the pixel focal length is calculated using the following formula:
[0065] f = F_mm x W_img / S_w_mm.
[0066] In this embodiment, the image center or the principal point obtained by camera calibration is taken as C = (cx, cy). When camera parameters are unavailable, an empirical focal length proportional to the input image resolution can be used as a backoff value.
[0067] S2, perform lightweight semantic segmentation on the input image: preprocess the document image to be processed, and input the preprocessed document image to be processed into the pre-trained lightweight semantic segmentation model to obtain the document foreground mask; the lightweight semantic segmentation model is configured to run locally on the mobile device;
[0068] Specifically, the data preprocessing of the document image to be processed includes: first, scaling the document image to be processed proportionally while maintaining the preset side length (scaling the maximum side to 512 pixels), and padding the edges to the input size of the lightweight semantic segmentation model (padding to a 512 x 512 image); then, converting the image to RGB format and normalizing it according to the mean and standard deviation during training.
[0069] In this embodiment, the lightweight semantic segmentation model adopts a binary classification model based on PP-MobileSeg; the binary classification model based on PP-MobileSeg is converted by ONNX and run locally on the mobile device in MNN format.
[0070] S3, Obtain candidate contours from document mask: Perform contour detection and convex hull processing on the document foreground mask to obtain multiple candidate contours, calculate the comprehensive score of each candidate contour, and select the candidate contour with the highest comprehensive score.
[0071] Specifically, the formula for calculating the overall score for each candidate contour is as follows:
[0072] Score = w1 x RectScore + w2 x RatioScore + w3 x AreaScore + w4 xCentralityScore
[0073] Wherein, Score represents the overall score, RectScore represents the ratio of the area of the candidate contour to the area of its minimum bounding rectangle, RatioScore represents the closeness of the aspect ratio of the candidate contour to the preset target interval, AreaScore represents the proportion of the candidate contour area to the effective image area, and CentralityScore represents the normalized score of the distance between the center of the candidate contour and the center of the image. w1, w2, w3 and w4 represent configurable weights.
[0074] S4, Selecting corner points from candidate contours: Perform multi-threshold polygon approximation on the candidate contour with the highest comprehensive score to obtain a quadrilateral. Restore the four points of the quadrilateral from the model coordinate system to the original image coordinate system, and sort them by polar angle based on the geometric center of the four points. Then rotate the top left corner to the first position to obtain ordered corner points arranged in the order of top left, top right, bottom right, and bottom left.
[0075] Specifically, for the candidate contour with the highest score, a quadrilateral is obtained by multi-threshold polygon approximation; if the number of vertices obtained by approximation is less than or greater than four, a minimum bounding rectangle backtracking is used. The four points are restored from the model coordinate system to the original image coordinate system, and then polar angle sorting is performed based on the geometric centers of the four points. The top left corner is rotated to the first position, resulting in four ordered corner points: top left P_tl, top right P_tr, bottom right P_br, and bottom left P_bl. Figure 3 As shown.
[0076] S5, Verify document quadrilaterals: Perform validity verification on ordered quadrilateral points. After the verification passes, expand the quadrilaterals by a finite number of pixels and constrain the expanded points within the range of the input image.
[0077] Specifically, this step first verifies the number, convexity, non-self-intersection, area ratio, and side length validity of the ordered four corner points. If the area is too small, too large, the side length is zero, or four valid corner points are not obtained, the original image is returned with a prompt to re-detect, or a rollback process without scaling compensation is initiated. Then, the boundary points of the verification results are expanded by a finite number of pixels, and the expanded points are constrained within the range of the input image.
[0078] S6, Calculate the original pixel aspect ratio: Based on the points constrained within the input image range, calculate the Euclidean lengths of the top, bottom, left, and right sides of the quadrilateral, and calculate the original pixel aspect ratio based on the average lengths of the top and bottom sides and the average lengths of the left and right sides.
[0079] Specifically, this step first calculates the Euclidean lengths L_top, L_bottom, L_left, and L_right of the quadrilateral's top, bottom, left, and right sides, respectively. Then, it calculates the average length of the top and bottom sides, and the average length of the left and right sides, using the following formulas:
[0080] L_h = (L_top + L_bottom) / 2
[0081] L_v = (L_left + L_right) / 2
[0082] Where L_h represents the average length of the top and bottom edges; L_v represents the average length of the left and right edges.
[0083] Finally, the original pixel aspect ratio is calculated based on the average lengths of the top and bottom edges and the average lengths of the left and right edges. The calculation formula is as follows:
[0084] A_raw = L_h / L_v
[0085] Where A_raw represents the original pixel aspect ratio, which only reflects the pixel ratio after being affected by perspective projection.
[0086] S7, calculate the directional shortening factor from the vanishing point and focal length: find the intersection of the upper line and the lower line to obtain the horizontal vanishing point, find the intersection of the left line and the right line to obtain the vertical vanishing point, and calculate the distance from the horizontal vanishing point and the vertical vanishing point to the principal point respectively. Combine the pixel focal length to calculate the perspective shortening factor in the width direction and the perspective shortening factor in the height direction.
[0087] Specifically, this step first finds the intersection of the upper and lower lines to obtain the horizontal vanishing point V_h, and finds the intersection of the left and right lines to obtain the vertical vanishing point V_v. Then, the distances d_h and d_v from the horizontal and vertical vanishing points V_h and V_v to the principal point C are calculated respectively.
[0088] Then, based on d_h, d_v, and the pixel focal length, the perspective shortening factors in the width and height directions are calculated. The calculation formulas are as follows:
[0089] k_w = d_h / sqrt(d_h^2 + f^2)
[0090] k_h = d_v / sqrt(d_v^2 + f^2)
[0091] Where k_w represents the perspective shortening factor in the width direction, d_h represents the distance from the vanishing point to the principal point in the horizontal direction, and f represents the pixel focal length; k_h represents the perspective shortening factor in the height direction, and d_v represents the distance from the vanishing point to the principal point in the vertical direction.
[0092] In this embodiment, the farther the vanishing point is from the principal point, the closer the perspective shortening factor is to 1; therefore, when the top and bottom edges are approximately parallel and the horizontal vanishing point cannot be stably calculated, the perspective shortening factor in the width direction is set to one; when the left and right edges are approximately parallel and the vertical vanishing point cannot be stably calculated, the perspective shortening factor in the height direction is set to one.
[0093] S8, Generate the target rectangle and perform perspective transformation: Calculate the restored physical aspect ratio based on the original pixel aspect ratio, the perspective shortening factor in the width direction, and the perspective shortening factor in the height direction, and limit the restored physical aspect ratio within a preset range; determine the output width and output height of the target rectangle based on the relationship between the average lengths of the top and bottom sides and the average lengths of the left and right sides of the quadrilateral, using the reference pixel count; construct a homography matrix with the ordered four corner points as the source points and the vertices of the target rectangle as the target points, and obtain the corrected rectangular document image after performing perspective transformation.
[0094] Specifically, this step first calculates the restored physical aspect ratio based on the original pixel aspect ratio, the perspective shortening factor in the width direction, and the perspective shortening factor in the height direction. The calculation formula is as follows:
[0095] A_corrected = A_raw x k_h / k_w
[0096] Wherein, A_corrected represents the restored physical aspect ratio; in order to suppress excessive magnification caused by extreme close-up, corner errors or abnormal vanishing points, this embodiment limits A_corrected to a preset range, for example: [0.5, 2.0].
[0097] Then, set the baseline number of pixels to B (e.g., 2400 pixels). When L_h is greater than L_v, W_out = B and H_out = B / A_corrected; when L_h is less than or equal to L_v, H_out = B and W_out = B x A_corrected.
[0098] Finally, construct the homography matrix H using the source point sequence [P_tl, P_tr, P_br, P_bl] and the vertices of the target rectangle, and execute H = getPerspectiveTransform(sourcePoints, destinationPoints) and I_out = warpPerspective(I, H, W_out, H_out) to output the corrected rectangular document image.
[0099] In the semantic segmentation-based mobile document edge detection and correction method provided in this embodiment, steps S1 to S5 constitute the main link of document edge detection; steps S6 to S8 use the ordered four corner points output by the main link to perform perspective ratio restoration.
[0100] The following section provides a specific example to further illustrate a method for mobile document edge detection and correction based on semantic segmentation provided in this embodiment.
[0101] Implementation environment:
[0102] Running on Android mobile devices: The device includes a camera, processor, memory, and display. The processor executes the application installed in memory to perform image acquisition, document detection, scale restoration, and perspective transformation. The detection library provides initialization, image processing, and release interfaces on the Android side. The application passes in the local object address of the OpenCV image matrix and the image width and height; the detection library returns an array of the four corner points of the document. The detection library loads the MNN inference engine and encrypted model data, performing inference on the device side.
[0103] Document edge detection example:
[0104] For mobile applications involving photographing documents, books, ID cards, and cards, a doc-card dataset was constructed, and a binary classification semantic segmentation model was trained. The final dataset contains 2797 labeled samples, with 1678, 559, and 560 samples for training, validation, and testing, respectively. The data_info.txt file retains image labels and polygon vertex coordinates; pixel-level segmentation labels corresponding to the original image are generated based on the polygon labels, where 0 represents background and 1 represents document foreground.
[0105] Specifically, the training list is recorded in the form of "original image path, grayscale label image path". The original image is in JPEG format, and the label image is a single-channel grayscale PNG image that corresponds to the original image pixel by pixel. The labeled scene covers mobile shooting targets such as documents, books, certificates and cards, and areas such as desktops, fingers, book spines, and borders are all treated as backgrounds.
[0106] The model uses PP-MobileSeg from PaddleSeg as its base network, with a training task category count of 2. A training configuration instance uses a 512 x 512 pixel input size, and undergoes normalization and data augmentation. After training, the model is exported to ONNX format, then converted to MNN format, and finally used by the MNN inference engine on Android devices for edge inference.
[0107] Specifically, during preprocessing, the image aspect ratio is maintained, the longest side is scaled to a preset length and padded with a constant value of 114; normalization uses parameters of 0.5 mean and 0.5 per channel, mapping pixel values from 0 to 255 to -1 to 1. The preprocessing process during inference is consistent with that during training. During the selection phase, instance segmentation and semantic segmentation were compared. Considering factors such as the size of the on-device installation package, inference latency, and deployment authorization, PP-MobileSeg was selected as the base network. During training, the training set is used to update network parameters, the validation set is used for model selection and parameter tuning, and an independent test set is used for final evaluation. Hyperparameters such as training epochs, batch size, and learning rate are set according to the PaddleSeg training configuration, and the final model version is determined by validation set metrics.
[0108] The final official verification results are shown in Table 1:
[0109] Table 1
[0110]
[0111] The aforementioned 559 formal validation results demonstrate that the lightweight semantic segmentation model provided in this embodiment can output a document foreground mask that can be used for subsequent contour filtering and corner point recovery. The independent test set contains 560 samples, which were used for final evaluation after the model version was determined. Figure 4 As shown.
[0112] Example of perspective proportion restoration:
[0113] Taking a 480 x 640 preview frame as an example, Camera2 acquires a pixel focal length of f = 345. After the detection library returns P_tl, P_tr, P_br, and P_bl, the lengths of the four sides are calculated to obtain L_h, L_v, and A_raw; the intersection points of two pairs of opposite sides are found to obtain V_h and V_v; d_h, d_v and k_w, k_h are calculated; the target aspect ratio is obtained according to A_corrected = A_raw x k_h / k_w; the target aspect ratio is restricted to [0.5, 2.0]; W_out and H_out are determined with a base side length of 2400 pixels; finally, the homography matrix is constructed and the perspective-corrected image is output.
[0114] In sample T6, which has been tested, the uncompensated ratio is 0.664, the shortening factor in the height direction is 0.992, the shortening factor in the width direction is 0.967, and the compensated ratio is 0.681. This sample demonstrates a situation where the degree of shortening differs in the two directions and the compensation direction can be positive or negative.
[0115] Actual operating results:
[0116] In an Android mobile application instance, the camera preview interface can detect documents in the captured image in real time and mark the detection result with a quadrilateral border after detecting a valid document boundary. Figure 5 This is a schematic diagram illustrating the document edge detection process. The cyan-green border represents the detected document boundaries, and the thumbnail in the lower right corner represents the recognized document image.
[0117] Perspective transformation effect:
[0118] Please see Figure 6 , Figure 6 This is a schematic diagram illustrating the effect of perspective transformation. The document in the input image has perspective distortion caused by the shooting angle; after obtaining the ordered four corner points, calculating the target rectangle, and performing homography transformation, the document boundaries in the output image are corrected to regular rectangles, facilitating subsequent reading, OCR, or PDF generation.
[0119] This embodiment provides a mobile document edge detection and correction method based on semantic segmentation, with the main technical concept being how to transform the document foreground mask output by a self-trained semantic segmentation model into reliable, ordered, and directly usable corner points for image transformation. Addressing the challenges of weak boundaries, complex backgrounds, and irregular document shapes in mobile images, the method first scales the input image proportionally and pads the edges, then outputs a document foreground mask from a lightweight semantic segmentation model. Subsequently, it performs external contour extraction, convex hull processing, comprehensive scoring of candidate quadrilaterals, adaptive polygon approximation, minimum bounding rectangle backtracking, inverse calculation of padding coordinates, boundary correction, and polar angle sorting to obtain corner points arranged in the order of top left, top right, bottom right, and bottom left. This method primarily solves the problem of mobile document edge detection struggling to stably output ordered corner points directly usable for cropping under conditions of weak contrast, complex backgrounds, shadows, reflections, and partial occlusion. Furthermore, based on the main detection chain, it utilizes vanishing point and camera focal length compensation to reduce aspect ratio deviations caused by simply flattening the document according to pixel side lengths.
[0120] The semantic segmentation-based mobile document edge detection and correction method provided in this embodiment has the following advantages:
[0121] (1) Outputting the foreground region of the document for weak boundary scenes: The self-trained semantic segmentation model outputs the pixel-level document region, without relying on a single grayscale threshold or local straight line, and can be used for mobile images with shadows, reflections, low-contrast paper and complex backgrounds.
[0122] (2) The mask is converted into an ordered quadrangular point that can be used directly: the contour convex hull, comprehensive scoring, adaptive approximation, circumscribed rectangle backtracking, coordinate inverse calculation and polar angle sorting constitute a continuous post-processing link. The output result is not only the segmented region, but also has a stable geometric point order, which can directly drive cropping, OCR and image transformation.
[0123] (3) Suitable for edge deployment: PP-MobileSeg lightweight networks can run locally on mobile devices such as Android / iOS after being converted by ONNX / MNN; contour filtering and corner point recovery use only limited geometric operations, avoiding uploading the original image to the server.
[0124] (4) Reduce manual cropping costs: When the detection confidence or geometric validity is insufficient, the process can revert to the original image or make manual adjustments instead of outputting disordered or self-intersecting cropping boxes.
[0125] (5) Provide a unified input for subsequent processing: the same ordered four-corner point sequence can be reused for preview border, automatic shooting judgment, perspective cropping, vanishing point ratio recovery, OCR and image to PDF conversion, reducing coordinate transformation errors between different processing modules.
[0126] (6) Stable segmentation performance on the formal validation set: The final doc-card dataset contains 2797 fully annotated samples, with 1678, 559, and 560 samples in the training, validation, and test sets, respectively. On 559 formal validation images, the model achieved mIoU 0.9232, document-foreground IoU 0.8494, Dice 0.9585, pixel accuracy 0.9970, and Kappa 0.9171.
[0127] In summary, the method provided in this embodiment can still stably detect document edges and output ordered four corner points under conditions of weak contrast, complex background, shadow, reflection and partial occlusion. At the same time, it restores the true physical aspect ratio of the paper by vanishing point and focal length compensation, making the perspective correction results more accurate.
[0128] Example 2
[0129] Please see Figure 7 This embodiment provides a mobile document edge detection and correction system based on semantic segmentation, including:
[0130] An image acquisition module is used to acquire the document image to be processed and the mobile camera parameters, and to calculate the pixel focal length based on the camera parameters; the camera parameters include the principal point;
[0131] The document segmentation module is used to preprocess the document image to be processed, and input the preprocessed document image to be processed into a pre-trained lightweight semantic segmentation model to obtain a document foreground mask; the lightweight semantic segmentation model is configured to run locally on the mobile device;
[0132] The four-corner point determination module is used to perform contour detection and convex hull processing on the document foreground mask to obtain multiple candidate contours, calculate the comprehensive score of each candidate contour, and select the candidate contour with the highest comprehensive score.
[0133] And perform multi-threshold polygon approximation on the candidate contour with the highest comprehensive score to obtain a quadrilateral. Restore the four points of the quadrilateral from the model coordinate system to the original image coordinate system, and sort them by polar angle based on the geometric center of the four points. Then rotate the upper left corner to the first position to obtain the ordered four corner points arranged in the order of upper left, upper right, lower right, and lower left.
[0134] The scaling recovery module is used to verify the validity of the ordered four corner points. After the verification is passed, the points are expanded outward by a finite number of pixels, and the expanded points are constrained within the range of the input image.
[0135] And based on the points constrained within the input image range, calculate the Euclidean lengths of the top, bottom, left, and right sides of the quadrilateral, and calculate the original pixel aspect ratio based on the average lengths of the top and bottom sides and the average lengths of the left and right sides.
[0136] The perspective transformation module is used to find the intersection of the upper and lower lines to obtain the horizontal vanishing point, and to find the intersection of the left and right lines to obtain the vertical vanishing point. It also calculates the distances from the horizontal and vertical vanishing points to the principal point, and calculates the perspective shortening coefficients in the width and height directions in combination with the pixel focal length.
[0137] The system calculates the restored physical aspect ratio based on the original pixel aspect ratio, the perspective shortening factor in the width direction, and the perspective shortening factor in the height direction, and limits the restored physical aspect ratio within a preset range; it determines the output width and output height of the target rectangle based on the average length of the top and bottom sides and the average length of the left and right sides of the quadrilateral, using the reference pixel count; it constructs a homography matrix using the ordered four corner points as source points and the vertices of the target rectangle as target points, and performs perspective transformation to obtain the corrected rectangular document image.
[0138] For details on the implementation of each module in a semantic segmentation-based mobile document edge detection and correction system, please refer to the above description of the limitations of a semantic segmentation-based mobile document edge detection and correction method, which will not be repeated here.
[0139] The technical features of the above embodiments can be combined in any way (as long as there is no contradiction in the combination of these technical features). For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; these embodiments not explicitly written should also be considered to be within the scope of this specification.
Claims
1. A method for edge detection and correction of mobile documents based on semantic segmentation, characterized in that, include: Step 1: Obtain the image of the document to be processed and the mobile camera parameters, and calculate the pixel focal length based on the camera parameters; The camera parameters include the principal point; Step 2: Perform data preprocessing on the document image to be processed, and input the preprocessed document image to be processed into a pre-trained lightweight semantic segmentation model to obtain the document foreground mask; the lightweight semantic segmentation model is configured to run locally on the mobile device; Step 3: Perform contour detection and convex hull processing on the document foreground mask to obtain multiple candidate contours, calculate the comprehensive score of each candidate contour, and select the candidate contour with the highest comprehensive score. Step 4: Perform multi-threshold polygon approximation on the candidate contour with the highest comprehensive score to obtain a quadrilateral. Restore the four points of the quadrilateral from the model coordinate system to the original image coordinate system, and sort them by polar angle based on the geometric center of the four points. Then rotate the upper left corner to the first position to obtain an ordered four corner points arranged in the order of upper left, upper right, lower right, and lower left. Step 5: Perform validity verification on the ordered four corner points. If the verification passes, expand the points by a finite number of pixels and constrain the expanded points within the range of the input image. Step 6: Based on the points constrained within the input image range, calculate the Euclidean lengths of the top, bottom, left, and right sides of the quadrilateral, and calculate the original pixel aspect ratio based on the average lengths of the top and bottom sides and the average lengths of the left and right sides. Step 7: Find the intersection of the upper and lower lines to obtain the horizontal vanishing point, and find the intersection of the left and right lines to obtain the vertical vanishing point. Calculate the distances from the horizontal and vertical vanishing points to the principal point, and combine the pixel focal length to calculate the perspective shortening coefficients in the width and height directions. Step 8: Calculate the restored physical aspect ratio based on the original pixel aspect ratio, the perspective shortening factor in the width direction, and the perspective shortening factor in the height direction, and limit the restored physical aspect ratio within a preset range; determine the output width and output height of the target rectangle based on the average length of the top and bottom sides and the average length of the left and right sides of the quadrilateral, using the reference pixel count; construct a homography matrix using the ordered four corner points as the source points and the vertices of the target rectangle as the target points, and perform perspective transformation to obtain the corrected rectangular document image.
2. The method for mobile document edge detection and correction based on semantic segmentation according to claim 1, characterized in that, In step 1, the camera parameters also include: physical focal length, sensor physical width, and image pixel width. The formula for calculating the pixel focal length is: f = F_mm x W_img / S_w_mm Where f represents the pixel focal length, F_mm represents the physical focal length, W_img represents the image pixel width, and S_w_mm represents the sensor physical width.
3. The method for mobile document edge detection and correction based on semantic segmentation according to claim 1, characterized in that, In step 2, the data preprocessing of the document image to be processed specifically includes: first, scaling the document image to be processed according to a preset side length while maintaining the ratio, and padding the sides to the input size of the lightweight semantic segmentation model; then, converting the image to RGB format and normalizing it according to the mean and standard deviation during training.
4. The method for mobile document edge detection and correction based on semantic segmentation according to claim 1, characterized in that, In step 2, the lightweight semantic segmentation model adopts a binary classification model based on PP-MobileSeg; the binary classification model based on PP-MobileSeg is converted by ONNX and run locally on the mobile device in MNN format.
5. The method for mobile document edge detection and correction based on semantic segmentation according to claim 1, characterized in that, In step 3, when calculating the comprehensive score for each candidate contour, the calculation formula is as follows: Score = w1 x RectScore + w2 x RatioScore + w3 x AreaScore + w4 xCentralityScore Wherein, Score represents the overall score, RectScore represents the ratio of the area of the candidate contour to the area of its minimum bounding rectangle, RatioScore represents the closeness of the aspect ratio of the candidate contour to the preset target interval, AreaScore represents the proportion of the candidate contour area to the effective image area, and CentralityScore represents the normalized score of the distance between the center of the candidate contour and the center of the image. w1, w2, w3 and w4 represent the weights, respectively.
6. The method for mobile document edge detection and correction based on semantic segmentation according to claim 1, characterized in that, In step 4, when performing multi-threshold polygon approximation on the candidate contour with the highest comprehensive score, if the number of vertices obtained from the approximation is less than or greater than four, then the quadrilateral is obtained by backtracking using the minimum bounding rectangle.
7. The method for mobile document edge detection and correction based on semantic segmentation according to claim 1, characterized in that, In step 5, the validity verification of the ordered four corner points specifically includes: verifying the number, convexity, non-self-intersection, area ratio, and side length of the ordered four corner points. If the area is too small, too large, the side length is zero, or four valid corner points are not obtained, the original image is returned and a prompt for re-detection is given, or a rollback process without proportional compensation is entered.
8. The method for mobile document edge detection and correction based on semantic segmentation according to claim 1, characterized in that, In step 7, when the top and bottom edges are approximately parallel and the vanishing point in the horizontal direction cannot be stably calculated, the perspective shortening factor in the width direction is set to one; when the left and right edges are approximately parallel and the vanishing point in the vertical direction cannot be stably calculated, the perspective shortening factor in the height direction is set to one.
9. The method for mobile document edge detection and correction based on semantic segmentation according to claim 1, characterized in that, In step 7, the formula for calculating the perspective shortening factor in the width direction is: k_w = d_h / sqrt(d_h^2 + f^2) Where k_w represents the perspective shortening factor in the width direction, d_h represents the distance from the vanishing point to the principal point in the horizontal direction, and f represents the pixel focal length; The formula for calculating the perspective shortening factor in the height direction is: k_h = d_v / sqrt(d_v^2 + f^2) Where k_h represents the perspective shortening factor in the height direction, and d_v represents the distance from the vanishing point in the vertical direction to the principal point.
10. A mobile document edge detection and correction system based on semantic segmentation, characterized in that, include: The image acquisition module is used to acquire the image of the document to be processed and the parameters of the mobile camera, and to calculate the pixel focal length based on the camera parameters; The camera parameters include the principal point; The document segmentation module is used to preprocess the document image to be processed, and input the preprocessed document image to be processed into a pre-trained lightweight semantic segmentation model to obtain a document foreground mask; the lightweight semantic segmentation model is configured to run locally on the mobile device; The four-corner point determination module is used to perform contour detection and convex hull processing on the document foreground mask to obtain multiple candidate contours, calculate the comprehensive score of each candidate contour, and select the candidate contour with the highest comprehensive score. And perform multi-threshold polygon approximation on the candidate contour with the highest comprehensive score to obtain a quadrilateral. Restore the four points of the quadrilateral from the model coordinate system to the original image coordinate system, and sort them by polar angle based on the geometric center of the four points. Then rotate the upper left corner to the first position to obtain the ordered four corner points arranged in the order of upper left, upper right, lower right, and lower left. The scaling recovery module is used to verify the validity of the ordered four corner points. After the verification is passed, the points are expanded outward by a finite number of pixels, and the expanded points are constrained within the range of the input image. And based on the points constrained within the input image range, calculate the Euclidean lengths of the top, bottom, left, and right sides of the quadrilateral, and calculate the original pixel aspect ratio based on the average lengths of the top and bottom sides and the average lengths of the left and right sides. The perspective transformation module is used to find the intersection of the upper and lower lines to obtain the horizontal vanishing point, and to find the intersection of the left and right lines to obtain the vertical vanishing point. It also calculates the distances from the horizontal and vertical vanishing points to the principal point, and calculates the perspective shortening coefficients in the width and height directions in combination with the pixel focal length. The system calculates the restored physical aspect ratio based on the original pixel aspect ratio, the perspective shortening factor in the width direction, and the perspective shortening factor in the height direction, and limits the restored physical aspect ratio within a preset range; it determines the output width and output height of the target rectangle based on the average length of the top and bottom sides and the average length of the left and right sides of the quadrilateral, using the reference pixel count; it constructs a homography matrix using the ordered four corner points as source points and the vertices of the target rectangle as target points, and performs perspective transformation to obtain the corrected rectangular document image.
