Method and apparatus for removing staple holes in a scanned image

Abstract

The application discloses a method and a device for removing a binding hole in a scanned image, and relates to the technical field of image processing. In the method, the device removes the binding hole in the scanned image based on a preset binding hole detection model to obtain a predicted position of the binding hole. In the case that the background plate of a scanner is gray, it is determined whether the minimum value of three channel color values of a pixel point of the scanned image is greater than a preset background brightness threshold value or whether the maximum value of the three channel color values is less than a preset threshold value. If yes, the pixel point of the scanned image is determined as a white point. If no, the pixel point of the scanned image is determined as a black point. The pixel point of the scanned image is converted into a white point or a black point to obtain a binary image. The predicted position of the binding hole in the binary image is cut into a sub-image. Contour searching is performed on the sub-image to obtain a binding hole mask image. The scanned image is filled and repaired based on the binding hole mask image and the color of the background plate of the scanner to obtain a target image with the binding hole removed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular, to a method and device for removing a binding hole in a scanned image. BACKGROUND

[0002] As a key image acquisition device, a scanner converts paper documents into digital image information through a photosensitive device. During the scanning process, if there is a binding hole on the scanned paper, the background plate of the scanner (usually black, white or gray) will be photographed through the hole, thereby forming a hole region consistent with the background color in the finally generated digital image. These holes not only have no actual information value, but also seriously affect the overall aesthetics and subsequent use of the image.

[0003] Some solutions based on heuristic rules are proposed in related technologies. A common technical solution is to first perform grayscale and binarization processing on the original color image to obtain a black-and-white binary image. Subsequently, on the basis of the binary image, the binding hole is searched and located through pre-set rules. These rules are usually based on assumptions about the physical characteristics of the binding hole, for example, through connected component analysis, regions that meet the pre-set size and shape (such as approximately circular) are selected as candidate binding holes. In order to further improve the accuracy, some solutions also combine position information, for example, it is assumed that the binding hole always appears near the edge of the paper, or multiple binding holes are roughly distributed on a straight line. In addition, some technical solutions introduce color features, and calculate the variance of pixel values and gradient values in the candidate region to assist in judgment, in order to distinguish between real binding holes and other dark content on the document.

[0004] However, related technologies are generally based on heuristic rules and rely on image binarization operation to segment the binding hole. This way can still work for regular binding holes on a black background, but when the scanner background is gray or the binding hole shape is complex (such as multiple binding holes overlapping), the pixel values of the hole region are very close to part of the document content (such as light-colored ink, seal), and related technologies are difficult to effectively separate them based on heuristic rules, reducing the accuracy of removing the binding hole in the scanned image. SUMMARY

[0005] The present application provides a method and device for removing a binding hole in a scanned image to improve the accuracy of removing the binding hole in the scanned image.

[0006] In a first aspect, a method and a device for removing a staple hole in a scanned image are provided. The method comprises: detecting, by the device, a scanned image to be processed based on a preset staple hole detection model to obtain a staple hole prediction position; determining, by the device, whether a minimum value of three-channel color values of a pixel point of the scanned image is greater than a preset background brightness threshold value or a maximum value of the three-channel color values is less than a preset threshold value in a case where a background plate of a scanner is gray; determining, by the device, that the pixel point of the scanned image is a white point if yes; determining, by the device, that the pixel point of the scanned image is a black point if no; converting, by the device, the pixel point of the scanned image into a white point or a black point to obtain a binary image; cutting, by the device, the staple hole prediction position in the binary image into a sub-image; performing, by the device, contour searching on the sub-image to obtain a staple hole mask image; and filling and repairing, by the device, the scanned image based on the staple hole mask image and a background plate color of the scanner to obtain a target image with the staple hole removed.

[0007] By adopting the above technical solution, the preset staple hole detection model can learn deep visual features of the staple hole from a large amount of data autonomously instead of relying on a preset simple geometric shape (such as a circle) or a color threshold value, so that the preset staple hole detection model can generalize and identify various irregular, damaged or special-shaped (such as square or elliptical) staple holes, thereby having stronger environmental adaptability. Based on the staple hole prediction position determined by the preset staple hole detection model, in the case of a gray background, whether the minimum value and the maximum value of the three-channel color values of the pixel fall outside a specific interval is judged, the gray hole region with a brightness between the paper and the dark text is identified as the foreground (black point), and the high-brightness paper background and the extremely dark text strokes and other contents are all classified as the background (white point), so that a binary image highlighting the staple hole can also be generated in a low-contrast scene, thereby improving the removal accuracy of the staple hole in the scanned image.

[0008] In combination with some embodiments of the first aspect, in some embodiments, the step of detecting, by the device, the scanned image to be processed based on the preset staple hole detection model to obtain the staple hole prediction position specifically comprises: performing, by the device, a standardization preprocessing operation on the scanned image to generate standardized image data; inputting, by the device, the standardized image data into the preset staple hole detection model to obtain an initial staple hole candidate region set and a confidence score corresponding to each initial staple hole candidate region; filtering, by the device, a staple hole candidate region with a confidence score less than a preset confidence threshold value in the initial staple hole candidate region set; and eliminating, by the device, a redundant staple hole candidate region with an overlap degree greater than a preset overlap threshold value from the filtered initial staple hole candidate region set based on a non-maximum suppression algorithm to obtain the staple hole prediction position.

[0009] By employing the above technical solution, the error caused by false detections and low-quality detection results is reduced by filtering out candidate binding hole regions with confidence scores lower than a preset confidence threshold, thereby improving the accuracy of binding hole prediction. A non-maximum suppression algorithm is used to process the filtered initial set of candidate binding hole regions, eliminating redundant candidate regions with overlap exceeding a preset overlap threshold, thus reducing redundant calculations and resource waste in subsequent processing.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the step of the removal device cropping the predicted binding hole position in the binarized image into a sub-image specifically includes: the removal device performing a morphological opening operation of a preset size on the binarized image; the removal device performing connected component analysis on the binarized image after the morphological opening operation to obtain all connected regions; the removal device calculating the area compactness parameter of each connected region; the removal device identifying connected regions with an area greater than a preset area threshold and a compactness within a preset range as potential binding hole regions; the removal device matching and verifying the position information of the potential binding hole regions with the predicted binding hole position to determine the final valid binding hole position; and the removal device cropping the binarized image based on the valid binding hole position to obtain a sub-image.

[0011] By employing the above technical solution, a morphological opening operation of a preset size is performed on the binarized image, breaking the subtle connections between the binding hole and surrounding text strokes or graphic elements. This avoids misidentifying the binding hole and document content as a single entity in subsequent connected component analysis, thereby improving the accuracy of binding hole region separation. Connected component analysis is performed on the processed binarized image to obtain all connected regions. Then, by calculating the area compactness parameter of each connected region, a geometric feature screening mechanism for the binding hole region is established. The condition that the area is greater than a preset area threshold reduces noise points and subtle interference, improving the accuracy of obtaining candidate binding hole regions. Matching and verifying the location information of potential binding hole regions with the predicted binding hole locations obtained from deep learning detection further improves the reliability of binding hole localization. Based on the effective binding hole locations, precise region cropping is performed on the binarized image to obtain sub-images, avoiding contour searching of the entire image and thus improving processing efficiency.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, the step of the removal device performing a morphological opening operation of a preset size on the binarized image specifically includes: the removal device calculating the optimal size of the morphological structural element based on the average size of the binding hole region in the predicted binding hole position; the removal device constructing an elliptical structural element of the optimal size; and the removal device using the elliptical structural element to sequentially perform morphological erosion and morphological dilation operations on the binarized image to break the fine connection between the binding hole and the text strokes.

[0013] Compared with the traditional rectangular or circular structure element, the elliptical structure element better matches the typical geometric shape features of the binding hole, can more accurately maintain the original contour features of the binding hole during the morphological operation, and reduces shape distortion. The erosion operation can disconnect the small connecting bridges between the binding hole and the text strokes, reduces the connection noise that may affect the accuracy of the connected component analysis, and the subsequent dilation operation can restore the original size and shape features of the main region of the binding hole, reduces the detection omission rate caused by excessive shrinkage of the binding hole region.

[0014] In some embodiments, the removing device performs contour searching on the sub-image to obtain the binding hole mask image, specifically including: the removing device extracts a coordinate point set of all closed contours in the sub-image based on an edge contour tracking algorithm; the removing device takes the closed contour with an enclosed area greater than a preset area threshold as a candidate binding hole contour; the removing device maps the coordinate points of the candidate binding hole contour from the coordinate system of the sub-image to the coordinate system of the scanned image; the removing device initializes a full-zero mask image with the same size as the scanned image; and the removing device marks the coordinate points of the candidate binding hole contour in the full-zero mask image as 255 to obtain the binding hole mask image.

[0015] In some embodiments, the removing device extracts a coordinate point set of all closed contours in the sub-image based on an edge contour tracking algorithm, provides complete boundary information and topological connection relationship, takes the closed contour with an enclosed area greater than a preset area threshold as a candidate binding hole contour, reduces the interference caused by noise points, small spots and other non-binding hole regions in the image, and improves the accuracy and reliability of the binding hole contour identification.

[0016] In some embodiments, the removing device fills and repairs the scanned image based on the binding hole mask image and the background plate color of the scanner to obtain a target image with the binding holes removed, specifically including: the removing device determines to use a specified color filling mode or a neighboring background color filling mode according to the filling mode parameter transmitted by the scanner device; in the case of using the specified color filling mode, the removing device replaces the pixel value of the scanned image with the pixel value of the pixel point corresponding to the position of the pixel with a pixel value of 255 in the binding hole mask image; in the case of using the neighboring background color filling mode, the removing device calculates the three-channel color mean value of the non-binding hole pixels in the ring-shaped region with a preset width around each binding hole; the removing device takes the three-channel color mean value as the filling color of the corresponding binding hole region; the removing device replaces the pixel value of the binding hole position in the scanned image based on the filling color; and the removing device performs edge smoothing processing on the target image after filling and repairing to eliminate the mutation between the filling region and the surrounding pixels, and generates the target image with the binding holes removed.

[0017] By adopting the technical scheme, in the designated color filling mode, the pixel points with a pixel value of 255 in the scanned image and the binding hole mask image are replaced with a preset color value, which is suitable for a scanned image with a relatively uniform background color, and can quickly complete the binding hole removal processing. In the near neighbor background color filling mode, the three-channel color mean value of the non-binding hole pixels in the ring-shaped area with a preset width around each binding hole is calculated as the filling color. This adaptive color calculation method can determine the most matched filling color value according to the actual background color characteristics around the binding hole, reduces the color difference problem that may be caused by the fixed color filling, and makes the repaired image more natural and harmonious in vision.

[0018] In combination with some embodiments of the first aspect, in some embodiments, after the step of removing the device performing edge smoothing processing on the target image filled and repaired, the method further comprises: the removing device extracting boundary pixels of each binding hole region in the target image; the removing device calculating a color difference measure value between each boundary pixel and its adjacent non-binding hole pixel; the removing device taking the boundary pixel with a color difference measure value greater than a preset difference threshold as a boundary pixel to be optimized; the removing device performing multi-scale Gaussian filtering processing on the boundary pixel to be optimized to generate candidate filling color values with different blur degrees; the removing device establishing a rectangular neighborhood window with a preset pixel radius centered on the boundary pixel to be optimized; the removing device extracting effective pixels in the rectangular neighborhood window except the binding hole region as a texture reference pixel set; and the removing device selecting a color value most matched with the texture features of the texture reference pixel set from the candidate filling color values for secondary filling.

[0019] By adopting the technical scheme, the removing device takes the boundary pixel with a color difference measure value greater than a preset difference threshold as a boundary pixel to be optimized, without the need to repeatedly process the entire repaired region, thereby improving the efficiency and pertinence of the optimization processing. The multi-scale Gaussian filtering processing on the boundary pixel to be optimized generates candidate filling color values with different blur degrees, providing a diversified color selection scheme, and obtaining various color fusion effects from fine to coarse through filtering processing of different scales, thereby providing rich candidate options for optimal color matching. The color value most matched with the texture features of the texture reference pixel set is selected from the candidate filling color values for secondary filling, so that the final repair effect is highly consistent with the surrounding area in both color and texture dimensions. This multi-level fine optimization processing mechanism improves the visual quality of the binding hole removal and improves the quality of the final target image.

[0020] In a second aspect, the embodiments of the present application provide a removing device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is configured to store computer program codes, the computer program codes comprising computer instructions, and the one or more processors are configured to invoke the computer instructions to cause the removing device to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0021] In a third aspect, the embodiments of the present application provide a computer program product comprising instructions which, when executed on a removing device, cause the removing device to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0022] In a fourth aspect, the embodiments of the present application provide a computer-readable storage medium comprising instructions which, when executed on a removing device, cause the removing device to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0023] It can be understood that the removing device provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the method provided in the embodiments of the present application. Therefore, the beneficial effects that can be achieved by the removing device, the computer program product, and the computer storage medium are referable to the beneficial effects in the corresponding method, which will not be described here again.

[0024] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0025] 1. The preset binding hole detection model can autonomously learn the deep visual features of the binding hole from a large amount of data, rather than relying on a preset simple geometric shape (such as a circle) or a color threshold, so that the preset binding hole detection model can generalize and identify various irregular, damaged or special-shaped (such as square, oval) binding holes, thereby having stronger environmental adaptability. On the basis of the binding hole prediction position determined by the preset binding hole detection model, for the gray background, by judging whether the minimum and maximum values of the pixel three-channel color value fall outside a specific interval, the gray hole region with brightness between the paper and the dark text is identified as the foreground (black point), and at the same time, the high-brightness paper background and the extremely dark text strokes and other contents are all classified as the background (white point), so that a binary image highlighting the binding hole can also be generated in a low-contrast scene, thereby improving the removal accuracy of the binding hole in the scanned image.

[0026] 2、The removal device extracts the coordinate point set of all closed contours in the sub-image based on the edge contour tracking algorithm, provides complete boundary information and topological connection relationship, takes the closed contour with an enclosed area greater than a preset area threshold as a screening mechanism of candidate binding hole contour, reduces the interference caused by noise points, small spots and other non-binding hole regions in the image, and improves the accuracy and reliability of the binding hole contour recognition.

[0027] 3、The removal device takes the boundary pixel with a color difference metric value greater than a preset difference threshold as a to-be-optimized boundary pixel, does not need to repeatedly process the entire repair region, improves the efficiency and pertinence of the optimization processing, performs multi-scale Gaussian filtering processing on the to-be-optimized boundary pixel to generate candidate filling color values with different blur degrees, provides diversified color selection schemes, obtains various color fusion effects from fine to coarse through filtering processing of different scales, provides rich candidate options for optimal color matching, selects a color value that is most matched with the texture features of the texture reference pixel set from the candidate filling color values for secondary filling, and makes the final repair effect highly consistent with the surrounding area in both color and texture dimensions. This multi-level fine optimization processing mechanism improves the visual quality of the binding hole removal and improves the quality of the final target image. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a flowchart of a method for removing a binding hole in a scanned image in an embodiment of the present application.

[0029] Figure 2 is a schematic diagram of an entity device structure of a removal device in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be limiting on the present application. As used in the specification and the appended claims of the present application, the singular forms “a,” “an,” and “the” are intended to include plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “and / or” used in the present application, mean and include any or all possible combinations of one or more listed items.

[0031] Hereinafter, the terms “first” and “second” are only for the purpose of description, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with “first” and “second” can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, the meaning of “a plurality of” is two or more, unless otherwise specified.

[0032] The application provides a method and a device for removing a staple hole in a scanned image, so as to improve the accuracy of removing the staple hole in the scanned image.

[0033] Please refer to Figure 1 A flowchart of a method for removing a staple hole in a scanned image is provided in the embodiments of the application.

[0034] S101, the removing device detects a scanned image to be processed based on a preset staple hole detection model, and obtains a staple hole prediction position.

[0035] The removing device refers to a computing device specially used for processing the task of removing the staple hole in the scanned image. The preset staple hole detection model represents a deep learning model trained by a large amount of data and capable of identifying the staple hole, such as a target detection model (such as YOLO) based on a convolutional neural network (CNN). The model can learn the visual features of the staple hole from a complex image background and generalize the visual features, and is not limited to a specific geometric shape. The scanned image to be processed refers to a digital document image containing the staple hole collected by a scanner device. The staple hole prediction position is used to represent the coordinate information of the staple hole in the image output by the detection model, in the form of a rectangular boundary box.

[0036] Specifically, first, the removing device will perform standardization preprocessing on the input scanned image, which includes scaling the image size to the standard size used during model training (for example, 640x640 pixels), and normalizing the pixel value (for example, converting from the [0, 255] integer range to the [0, 1] floating-point number range) to improve the stability of the model input. Then, the standardized image data is input into the preset staple hole detection model for inference calculation. The preset staple hole detection model outputs an initial candidate box and a confidence score corresponding to each box, which represents the probability that the model considers the box to be a staple hole. Then, in order to filter out most of the invalid low-quality candidate boxes, the device discards all candidate boxes with a confidence score lower than a preset threshold (for example, 0.5). Finally, since a single staple hole may be detected by multiple highly overlapping candidate boxes, the removing device performs a non-maximum suppression (NMS) algorithm: among the overlapping boxes, the one with the highest confidence is retained, and other redundant boxes overlapping with it by more than a preset threshold (for example, 0.45) are removed, and finally a set of staple hole prediction positions is obtained.

[0037] In some embodiments, the construction and deployment of the binding hole detection model can be implemented in various ways: optionally, first, collect scanning image samples containing various shapes and sizes of binding holes, use label tools such as labelImg to label the binding holes with bounding boxes, then divide the labeled data into training set and test set according to the ratio of 8:2, train the YOLO model based on the PyTorch deep learning framework, and convert the PyTorch format model into ONNX format after training for easy deployment. Optionally, a pre-trained general object detection model can also be used as a basis to fine-tune on the binding hole dataset through transfer learning. First, freeze the backbone network parameters of the model, only train the detection head part, then gradually unfreeze more layers for end-to-end training, and finally optimize the model performance through model quantization and pruning techniques to meet real-time processing requirements.

[0038] S102, in the case of a gray background plate of the scanner, the device determines whether the minimum value of the three-channel color value of the pixel points of the scanning image is greater than the preset background brightness threshold or the maximum value of the three-channel color value is less than the preset threshold.

[0039] The background plate of the scanner refers to the bottom plate structure opposite the photosensitive device inside the scanner device, which provides a background for the scanned document. A gray background plate means that the color of the bottom plate is gray. Compared with a black or white background, the binding hole and the document content are closer in pixel value on a gray background, increasing the difficulty of separation. The three-channel color value refers to the numerical value of the blue, green, and red color components of each pixel point in a color image. The preset background brightness threshold is pre-set according to the characteristics of the background plate of the scanner and can be set to the maximum brightness value of the background color. For example, the normal background plate color is between 130 and 140 brightness. In an exemplary embodiment, the preset background brightness threshold can be set to 140. The preset threshold value represents a fixed numerical value for determining whether a pixel point is dark content. In an exemplary embodiment, it can be set to 40.

[0040] Specifically, this step is performed after obtaining the predicted position of the binding hole. The fundamental purpose is not to separate the content from the background as in the traditional method, but to identify the binding hole as the foreground (black point) in a complex environment (containing white paper, dark text, gray hole, possible light content, and possible stains), and to classify all other elements (paper and text, etc.) as the background (white point). If the minimum channel value of a pixel point is greater than the preset background brightness threshold, it is a high-light area such as white paper or very light stains, which should be determined as a white point. If the maximum channel value of a pixel point is darker than the preset threshold, it is a very dark area such as black text strokes, which should be determined as a white point. Only when a pixel point is neither too bright nor too dark, i.e. the minimum value of the three channel color values of a pixel point in the scanned image is not greater than the preset background brightness threshold, and the maximum value of the three channel color values is not less than the preset threshold 40, the pixel is determined as a black point. The feature that the brightness falls exactly in the middle is the typical feature of the binding hole area (including its shadow) on the gray background plate.

[0041] In some embodiments, adaptive pixel classification in a gray background can be achieved in various ways: optionally, first, color statistical analysis is performed on the entire scanned image, the distribution histogram of gray values in the image is calculated, the main distribution interval of the background gray is identified by a peak detection algorithm, then the preset background brightness threshold is dynamically adjusted based on the statistical results, and finally the adjusted threshold is applied to each pixel point for classification judgment. Optionally, a local adaptive method can also be used, the image is divided into multiple sub-regions, the local background brightness features of each sub-region are calculated respectively, the corresponding judgment threshold is set for each sub-region according to the local features, and then the black and white classification of the pixel points is independently performed in each sub-region. Finally, the results of all sub-regions are combined to form a complete classification result.

[0042] S103, the removal device determines that the pixel point of the scanned image is a white point.

[0043] Specifically, this step is performed when the condition judgment of step S102 is yes. When the removal device detects that the minimum value of the three channel color values of a pixel point in the scanned image is greater than the preset background brightness threshold, or the maximum value of the three channel color values is less than the preset threshold, the removal device identifies and marks the pixel point as a white point.

[0044] S104, the removal device determines that the pixel point of the scanned image is a black point.

[0045] Specifically, this step is performed when the condition judgment of step S102 is no. When the removal device detects that the minimum value of the three channel color values of a pixel point in the scanned image is not greater than the preset background brightness threshold, and the maximum value of the three channel color values is not less than the preset threshold 40, the removal device identifies and marks the pixel point as a black point.

[0046] S105, the removing device converts the pixel points of the scanned image into white points or black points to obtain a binary image.

[0047] The pixel point conversion refers to a process of converting each pixel point in the original scanned image into a binary representation according to the aforementioned classification result. The binary image refers to a single-channel grayscale image containing only two pixel values (0 and 255), which is used to simplify subsequent contour detection and morphological processing.

[0048] Specifically, this step is executed after the black and white classification of all pixel points is completed. When the background is gray, the removing device traverses each pixel point in the scanned image, and according to the classification results of steps S103 and S104, sets all pixels marked as black points to 0 (black foreground) in the binary image, and sets all pixels marked as white points to 255 (white background) in the binary image. This conversion process creates a binary image that is exactly the same size as the original scanned image but contains only black and white colors. In this binary image, the binding holes are presented in black regions, while the background regions are presented in white, providing a clear foreground-background separation effect for subsequent morphological processing and contour finding.

[0049] In some embodiments, when the scanner background is black, the removing device can call a standard OTSU binary function to process the entire grayscale scanned image to obtain a global binary image. In this high-contrast scenario, OTSU can well separate white paper from black background (and holes).

[0050] S106, the removing device cuts the binding hole prediction position in the binary image into a sub-image.

[0051] The binding hole prediction position refers to the rectangular bounding box coordinate information of the binding hole output by the YOLO detection model in step S101. The sub-image represents a local image region containing a single or multiple binding holes cut from the binary image.

[0052] Specifically, this step is performed after obtaining the binary image. The removal device first calculates the most suitable elliptical structure element size based on the average size of the detected staple hole region in step S101, and then constructs the elliptical structure element to perform a morphological opening operation on the binary image. By first eroding and then dilating, the small connections between the staple holes and the surrounding text strokes are disconnected, while the integrity of the main body of the staple hole is maintained. Next, the removal device performs connected component analysis on the processed binary image, identifies all connected foreground pixel regions, and calculates the area compactness parameter (also known as circularity or shape factor) of each connected region. For a standard circle, the area compactness parameter is 1, and the more irregular the shape, the smaller the area compactness parameter. By setting a pre-set compactness range (e.g. [0.6, 1.0]), regions that are more like staple holes in shape can be retained, while excluding long, irregular residual text strokes or interference. The removal device screens potential staple hole regions from all connected regions based on the screening conditions that the area is greater than a pre-set threshold (e.g. 1000 pixels) and the compactness is within the pre-set compactness range. Subsequently, the removal device spatially matches the position information of these potential staple hole regions with the YOLO detection position obtained in step S101. Only those connected regions that are both inside or near the YOLO prediction box and meet the morphological and geometric screening conditions are confirmed as the final effective staple hole positions, and the binary image is cut based on this to obtain a sub-image.

[0053] In some embodiments, the cutting of the staple hole region can be achieved in several ways. Optionally, a multi-scale morphological processing can be used. After obtaining the predicted staple hole locations (i.e., a series of rectangular bounding boxes) in S101, the removal device extracts the width and height of each predicted staple hole rectangle. Then, the average width and average height are obtained by taking the arithmetic mean of all the width and height values. The size of the morphological structuring element is critical: too small and it can not effectively disconnect the text from the staple hole; too large and it can over-erode the staple hole itself, resulting in distorted or lost contours. Therefore, an optimal size should be positively correlated with but slightly smaller than the average size of the staple holes. For example, in a specific implementation, the optimal size can be set to a pre-set proportion of the average size, such as setting the optimal width to 0.7 times the average width and the optimal height to 0.7 times the average height. Then, based on the optimal width and the optimal height, the removal device calls an image processing library (e.g., the getStructuringElement function in OpenCV) and specifies the shape parameter as MORPH_ELLIPSE, thereby generating an elliptical structuring element. The elliptical shape is chosen because it can better approximate the geometry of most staple holes. Finally, the removal device uses the elliptical structuring element to sequentially perform morphological erosion and morphological dilation operations on the binary image. The erosion operation shrinks the foreground region (black region) and can erode away small text strokes or noise points connected to the edges of the staple hole. The subsequent dilation operation restores the size of the staple hole body after erosion, but the small connections that were disconnected earlier will not be reconnected. Through this series of operations, the small connections between the staple hole and the text strokes are disconnected while preserving the integrity of the staple hole body. As another optional implementation, a multi-scale morphological processing can be used. The removal device can pre-construct a plurality of elliptical structuring elements of different sizes (e.g., sizes increasing from small to large) and use them to perform opening operations on the binary image. Then, the processing results at each scale are independently analyzed for connected components and features. By comparing the stability of the target connected components at different scales (e.g., the area and compactness of the features change the least at which scale), an optimal processing scale is automatically selected. Based on the processing results at the optimal scale, subsequent cutting of the staple hole region is performed. This method can better adapt to situations where there are multiple staple holes of significantly different sizes in the image.

[0054] S107, the removal device performs contour finding on the sub-image to obtain a staple hole mask image.

[0055] The contour finding refers to a computer vision algorithm that detects and extracts the boundary of a target object in a binary image. The binding hole mask image represents a binary image with the same size as the original scanned image, in which the pixel value of the binding hole region is 255 and the pixel value of the background region is 0.

[0056] Specifically, this step is performed after obtaining the sub-image. The removal device uses an edge contour tracking algorithm to perform contour detection on the sub-image. The algorithm identifies and tracks the connection relationship between adjacent pixels by scanning the foreground pixels in the binary image, and extracts the complete coordinate point set of all closed contours. In order to exclude noise and small area interference regions, the removal device calculates the enclosed area of each closed contour, and screens the contours with an area greater than a preset threshold as candidate binding hole contours. The threshold is usually set according to the typical size of the binding hole. Since the sub-image is a local region cut from the original scanned image, the removal device needs to map the coordinate points of the candidate contours from the local coordinate system of the sub-image back to the global coordinate system of the scanned image, which is achieved by adding the offset of the sub-image in the scanned image to the original coordinates. Finally, the removal device creates a mask image with all pixel values being 0, which has the same size as the scanned image, and marks the mapped candidate binding hole contour coordinates as 255 in the mask image to form the binding hole mask image.

[0057] In some embodiments, the extraction of the contour and the generation of the mask can be implemented in various ways. Optionally, the removal device uses a multi-level contour detection strategy. First, the RETR_EXTERNAL mode is used to extract the outermost contour to avoid internal hole interference. Then, the RETR_LIST mode is used to extract all levels of contour information for each outer contour. The true binding hole boundary is identified through contour level relationship analysis. Next, the Douglas-Peucker algorithm is used to simplify the number of coordinates of the contour through polygon approximation. Finally, the simplified contour is completely filled into the mask image through a contour filling algorithm. Optionally, a contour quality evaluation method can also be used. The removal device not only considers the contour area but also comprehensively evaluates the geometric features such as the circularity, convexity and regularity of the contour. The minimum circumscribed circle and the minimum circumscribed rectangle of the contour are calculated. The contour quality is judged by comparing the ratio of the contour area to the circumscribed figure area. Only the contour with a quality score exceeding a preset threshold is recognized as an effective binding hole contour. Then, an anti-aliasing filling technique is used to draw the contour region in the mask image.

[0058] S108, the removal device fills and repairs the scanned image based on the binding hole mask image and the background plate color of the scanner to obtain a target image with the binding holes removed.

[0059] The filling repair refers to an image repair process of replacing pixel values in the binding hole area with specific color or texture information. The background board color represents the color characteristics of the physical background board of the scanner device, and is used to guide the color filling of the binding hole area.

[0060] Specifically, this step is performed after obtaining the binding hole mask image. The removal device first determines the specific repair strategy according to the filling mode parameter transmitted by the scanner device. In the specified color filling mode, the removal device directly replaces the pixel position marked as 255 in the binding hole mask image in the scanned image with a preset color value, which is usually set as an RGB value matching the color of the scanner background board. In the near-neighbor background color filling mode, the removal device adopts a more intelligent adaptive filling strategy: for each detected binding hole area, the removal device calculates the RGB three-channel color mean value of all non-binding hole pixels in the annular area of a predetermined width (such as 20-30 pixels) around the binding hole area, and uses the mean value as the special filling color of the corresponding binding hole area, and then replaces the pixel values at the binding hole position based on the calculated filling color. After completing the basic filling, the removal device performs edge smoothing on the repair area to eliminate the color discontinuity between the filled area and the surrounding pixels through Gaussian filtering or bilateral filtering technology, and improves the naturalness of the repair effect. In order to further improve the repair quality, the removal device also performs secondary optimization processing: extracting the boundary pixels of each binding hole area, calculating the color difference measure value of the boundary pixels and the adjacent non-binding hole pixels, performing multi-scale Gaussian filtering on the boundary pixels whose difference value exceeds the preset threshold to generate candidate filling color values, and then establishing a rectangular neighborhood window to extract a set of texture reference pixels, and finally selecting the color value that best matches the texture features for fine secondary filling.

[0061] In some embodiments, the quality of the binding hole filling repair can be improved in various ways: optionally, the removal device can adopt a layered repair strategy, first expand the binding hole mask area using morphological closing operation to ensure complete coverage of the binding hole edge, then divide the filling process into two stages of core filling and edge transition, use uniform filling color for the core area, use gradient transition technology for the edge area, and finally apply anisotropic diffusion filtering to optimize the texture consistency of the entire repair area. Optionally, a texture synthesis method based on samples can also be used, and the removal device establishes multiple sampling windows around the binding hole to extract background texture samples, uses a texture synthesis algorithm to generate filling content consistent with the texture pattern of the surrounding area, and seamlessly integrates the synthesized texture into the original image through Poisson image editing technology, to achieve a more natural repair effect in vision.

[0062] In the above embodiment, the preset binding hole detection model quickly locates the approximate position of the binding hole in the scanned image, and improves the processing efficiency of extracting the position of the binding hole. For the specific application scenario of the gray background plate of the scanner, a special binary judgment logic is established by analyzing the relationship between the minimum value and the maximum value of the three-channel color value of the pixel point and the preset threshold. Compared with the traditional fixed threshold binary method, this binary judgment logic has stronger environmental adaptability and higher segmentation accuracy. By cutting the predicted position of the binding hole in the binary image into a sub-image, the interference of irrelevant areas is reduced, and the accuracy of contour finding is improved. Based on the filling repair strategy of the binding hole mask image and the background plate color of the scanner, the final target image is close to the original scanning effect without the binding hole in terms of visual effect.

[0063] The above describes a method for removing a binding hole in a scanned image according to an embodiment of the present application. Next, an exemplary removal device 200 provided by an embodiment of the present application is introduced.

[0064] Figure 2 FIG. 1 is an exemplary hardware structure schematic diagram of the removal device 200 provided by an embodiment of the present application. In some embodiments, the removal device 200 is a computer device. The computer device includes a processor, a memory, and a network interface connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with other terminals or servers outside through network connection. In some embodiments, the network interface can be a wired network interface, and in some embodiments, the network interface can also be a wireless network interface. The computer program is executed by the processor to implement a method for removing a binding hole in a scanned image according to an embodiment of the present application.

[0065] Those skilled in the art can understand that, Figure 2 The structure shown in FIG. 1 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0066] In some embodiments of the present application, a computer-readable storage medium is also provided, including instructions that, when executed on the removal device 200, can cause the removal device 200 to perform a method for removing a binding hole in a scanned image according to an embodiment of the present application.

[0067] The above-described embodiments are merely intended to illustrate the technical solutions of the present application, but not to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those ordinarily skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or equivalently replace some of the technical features thereof; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0068] In the above-described embodiments, the term "when" can be interpreted as meaning "if" or "after" or "in response to determining" or "in response to detecting" according to the context. Similarly, the phrase "upon determining" or "if detecting (the stated condition or event)" can be interpreted as meaning "if determining" or "in response to determining" or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)" according to the context.

[0069] In the above-described embodiments, all or some of the flowcharts or functional modules can be realized in the form of a software program, and can be stored in any nonvolatile storage medium (for example, a floppy disk, a hard disk, a DVD, a CD, a ROM, a RAM, or a memory stick) or can be transmitted over a network. The software program can include a plurality of different programs, and can be a single program or a plurality of programs. The software program can be stored and provided in a file in any form, such as an executable file, a library, an object file, a script file, or a source file, and can be stored and provided in a compressed or unzipped format. The software program can be stored and provided in a form of a computer program product, such as a floppy disk, a hard disk, a DVD, a CD, a ROM, a RAM, or a memory stick, or can be transmitted over a network. The computer program product can be distributed over a network, such as the Internet, a local area network, a metropolitan area network, and a wide area network, using a plant, a communication line, a wireless channel, a wireless communication protocol, or a wired communication protocol.

[0070] Those ordinarily skilled in the art can understand that all or some of the flowcharts in the above-described embodiments can be implemented by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium and can include the flowcharts of the above-described embodiments when executed. The aforementioned storage medium includes a ROM or a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A method for removing binding holes from a scanned image, characterized in that, Applied to a removal device, the method includes: The removal device detects the scanned image to be processed based on a preset binding hole detection model to obtain the predicted position of the binding hole. When the background of the scanner is gray, the removal device determines whether the minimum value of the three-channel color value of the pixel in the scanned image is greater than a preset background brightness threshold or whether the maximum value of the three-channel color value is less than a preset threshold. If so, the removal device determines that the pixel of the scanned image is a white point; If not, the removal device determines that the pixels in the scanned image are black dots; The removal device converts the pixels of the scanned image into black or white dots to obtain a binarized image. The removal device crops the predicted position of the binding hole in the binarized image into a sub-image; The removal device performs contour search on the sub-image to obtain a binding hole mask image; The removal device fills and repairs the scanned image based on the binding hole mask image and the background color of the scanner to obtain a target image with the binding holes removed.

2. The method according to claim 1, characterized in that, The step of the removal device detecting the scanned image to be processed based on a preset binding hole detection model to obtain the predicted position of the binding hole specifically includes: The removal device performs a standardization preprocessing operation on the scanned image to generate standardized image data; The removal device inputs the standardized image data into the preset binding hole detection model to obtain an initial set of binding hole candidate regions and a confidence score corresponding to each initial binding hole candidate region. The removal device filters out binding hole candidate regions in the initial binding hole candidate region set whose confidence scores are less than a preset confidence threshold. The removal device eliminates redundant binding hole candidate regions with an overlap greater than a preset overlap threshold by using a non-maximum suppression algorithm on the filtered initial binding hole candidate region set, thereby obtaining the predicted binding hole position.

3. The method according to claim 1, characterized in that, The step of the removal device cropping the predicted binding hole position in the binarized image into a sub-image specifically includes: The removal device performs a morphological opening operation of a preset size on the binarized image; The removal device performs connected component analysis on the binarized image after morphological opening operation to obtain all connected regions. The removal device calculates the area compactness parameters of each connected region; The removal device uses connected regions with an area greater than a preset area threshold and a compactness within a preset range as potential binding hole regions. The removal device matches and verifies the location information of the potential binding hole area with the predicted binding hole location to determine the final valid binding hole location. The removal device performs region cropping on the binarized image based on the effective binding hole position to obtain the sub-image.

4. The method according to claim 3, characterized in that, The step of the removal device performing a morphological opening operation of a preset size on the binarized image specifically includes: The removal device calculates the optimal size of the morphological structural element based on the average size of the binding hole region in the predicted binding hole position; The removal device constructs the elliptical structural element of the optimal size; The removal device uses the elliptical structuring element to sequentially perform morphological erosion and morphological dilation operations on the binarized image, breaking the fine connection between the binding hole and the text strokes.

5. The method according to claim 1, characterized in that, The step of the removal device performing contour searching on the sub-image to obtain a binding hole mask image specifically includes: The removal device extracts the set of coordinate points of all closed contours in the sub-image based on an edge contour tracking algorithm; The removal device uses closed contours with an area larger than a preset area threshold as candidate binding hole contours. The removal device maps the coordinates of the candidate binding hole outline from the coordinate system of the sub-image back to the coordinate system of the scanned image; The removal device initializes a zero-mask image of the same size as the scanned image; The removal device marks the coordinates of the candidate binding hole outline in the all-zero mask image as 255, thus obtaining the binding hole mask image.

6. The method according to claim 1, characterized in that, The step of the removal device filling and repairing the scanned image based on the binding hole mask image and the background color of the scanner to obtain the target image with the binding holes removed specifically includes: The removal device determines whether to use a specified color fill mode or a neighboring background color fill mode based on the fill mode parameters transmitted from the scanner device. When a specified color fill mode is used, the removal device replaces the pixel value at the position corresponding to the pixel with a pixel value of 255 in the scanned image and the binding hole mask image with a preset color value. When using the nearest neighbor background color fill mode, the removal device calculates the three-channel color average of non-binding hole pixels within a preset width annular area around each binding hole; The removal device uses the average value of the three-channel colors as the fill color for the corresponding binding hole area; The removal device replaces pixel values ​​at the binding hole positions in the scanned image based on the fill color; The removal device performs edge smoothing processing on the target image after filling and repair, eliminating abrupt changes between the filled area and surrounding pixels, and generating a target image with the binding hole removed.

7. The method according to claim 6, characterized in that, After the removal device performs edge smoothing processing on the filled and repaired target image to eliminate abrupt changes between the filled area and surrounding pixels, generating a target image with the binding hole removed, the method further includes: The removal device extracts the boundary pixels of each binding hole region in the target image; The removal device calculates a color difference metric between each boundary pixel and its adjacent non-binding hole pixels. The removal device identifies boundary pixels whose color difference metric value is greater than a preset difference threshold as boundary pixels to be optimized. The removal device performs multi-scale Gaussian filtering on the boundary pixels to be optimized to generate candidate fill color values ​​with different degrees of blur. The removal device establishes a rectangular neighborhood window with a preset pixel radius centered on the boundary pixel to be optimized; The removal device extracts the effective pixels within the rectangular neighborhood window, excluding the binding hole area, as a texture reference pixel set. The removal device selects the color value that best matches the texture features of the texture reference pixel set from the candidate fill color values ​​for secondary filling.

8. A removal device, characterized in that, The removal apparatus includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and the one or more processors invoking the computer instructions to cause the removal apparatus to perform the method as described in any one of claims 1-7.

9. A computer program product containing instructions, characterized in that, When the computer program product is run on the removal device, the removal device performs the method as described in any one of claims 1-7.

10. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is executed on the removal device, the removal device performs the method as described in any one of claims 1-7.

Images