Picture correction method, device and page detection model building method

Abstract

The present application belongs to the technical field of image processing, and provides a picture correction method, device and computer equipment, which are used for identifying a subject area in a picture and correcting the subject area in the picture. The method comprises the following steps: taking four vertices of the picture as reference points, determining a plurality of positioning key points by regression calculation, wherein the positioning key points are used to determine the subject area of the picture; and identifying and correcting the subject area according to the positioning key points, so as to remove irrelevant information and normalize the subject area. The present application identifies a page area in a picture to be processed through an image positioning regression model, can more accurately identify the page area, can more accurately identify whether there is a page area in the picture to be processed, and solves the problem of low image recognition accuracy caused by factors such as inclination of the page area relative to the picture, irrelevant background, non-subject area and the like.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and is particularly applicable to the field of image correction technology. More specifically, it relates to an image correction method, a device, and a method for building a page detection model. Background Technology

[0002] Currently, an increasing number of application scenarios (such as online review and search queries) require uploading photos and images, and thus image recognition. For example, in the scenario of searching for answers by taking a picture, the photos taken by users often have issues such as tilting, missing corners, objects appearing larger in the foreground than in the background, having irrelevant backgrounds, or containing non-subject areas. These issues can negatively impact image recognition, and may even lead to recognition failure or inaccurate recognition. Therefore, image correction is necessary.

[0003] In related technologies, a method for correcting image blur is disclosed. This method includes: acquiring an image taken by a camera under a photo-based question-and-answer function; acquiring the blur distance of character strokes in the image; and reducing the blur distance to enhance the clarity of the character strokes in the image. This method solves the image blur problem by adding blur correction to the captured image during the photo-taking process, reducing the blur distance, and thus enhancing the clarity of the character strokes in the image. This increases the visibility of characters in the image and reduces the difficulty of question recognition and extraction in the photo-based question-and-answer function. Furthermore, in existing technologies, for photo-based question-and-answer applications, image segmentation methods are typically used to segment the main question region. However, this method suffers from slow speed and unstable segmentation edges. Therefore, there is still significant room for improvement in image recognition, image classification, and image segmentation due to factors such as image tilt, irrelevant backgrounds, and non-subject areas.

[0004] Therefore, it is necessary to provide an image correction method to solve the above problems. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] The present invention aims to at least solve the technical problems of low accuracy in identifying the main body region of an image, inability to accurately locate the main body region in an image, and insufficient precision in removing irrelevant information from an image.

[0007] (II) Technical Solution

[0008] To address the aforementioned technical problems, one aspect of the present invention proposes an image correction method for identifying and correcting the main body region in an image. The method comprises the following steps: using the four vertices of the image as reference points, employing regression calculation to determine several key positioning points, which are used to determine the main body region of the image; and identifying and correcting the main body region based on the key positioning points to remove irrelevant information and standardize the main body region.

[0009] According to an optional embodiment of the present invention, the main area of ​​the image includes a page area, and the plurality of positioning key points include the four vertices of the page.

[0010] According to an optional embodiment of the present invention, the step of determining several key positioning points by regression calculation includes: establishing an image positioning regression model and using the trained image positioning regression model to obtain the position information of the key positioning points of the image.

[0011] According to an optional embodiment of the present invention, the establishment of the image localization regression model includes labeling sample data and using the sample data to train the image localization regression model, wherein the labeled sample data refers to the labeling of sample images to mark the localization key points of the main body region in the sample images and whether the main body region is included.

[0012] According to an optional embodiment of the present invention, the image localization regression model takes an image to be processed as input and outputs the location coordinates and confidence level of the localization key points of the main body region in the image. The confidence level represents the reliability probability of the output localization key point coordinates. Optionally, when the confidence level is less than a predetermined threshold, the correction of the main body region is stopped and a warning message of no main body region is output.

[0013] According to an optional embodiment of the present invention, when the main area of ​​the image is a page area, the annotation of the sample image includes: when the page area of ​​the sample image is entirely within the sample image, directly annotating the four vertices of the page as the positioning key points; when the page area of ​​the sample image is partially outside the sample image, calculating the positions of the page vertices outside the sample image based on the page vertices within the sample image and the intersection of the page edge and the image edge, and annotating the page vertices within the image and the calculated page vertices outside the sample image as the positioning key points.

[0014] Optionally, calculating the position of page vertices outside the sample image includes: marking the vertices within the sample image and the intersection of the page edge and the image edge; traversing all marked points of the sample image, sequentially determining whether two adjacent marked points are simultaneously located on the same image edge, wherein the marked points include the vertices within the image and the intersection points; when it is determined that the two adjacent marked points are simultaneously located on the same image edge, further determining whether the included angle formed by the intersection of the extended edges of the two page regions where the two marked points are located satisfies a first threshold angle; when the included angle satisfies a first threshold angle, marking the two adjacent marked points as non-vertices, and marking the intersection formed by the extended edges of the two page regions corresponding to the two non-vertices as the predicted vertex of the page region outside the image, while deleting the two marked non-vertices, finally obtaining a sample image marked with four positioning key points; optionally, the first angle threshold is 60 degrees.

[0015] According to an optional embodiment of the present invention, the correction of the main body region includes: performing radial transformation and / or orientation correction on the main body region based on the positioning key points to obtain a corrected image.

[0016] A second aspect of the present invention also provides a method for building a page detection model, the page detection model being used to detect the four vertices of a page region of an image, the method comprising the following steps: building a keypoint detection model; training the keypoint detection model using an image labeled with the four vertices of the page region, such that the trained keypoint detection model can output the four vertices of the page region and their confidence scores; wherein the image labeled with the four vertices of the page region is obtained through the aforementioned step of labeling sample images.

[0017] A third aspect of the present invention also provides an image correction method, the method comprising the following steps: inputting an image to be processed into a page detection model generated by the page detection model construction method according to the second aspect of the present invention, obtaining the four vertices and confidence scores of the page region of the image to be processed; determining whether the confidence score is greater than or equal to a first threshold; when the confidence score is greater than the first threshold, performing page region identification and correction on the image to be processed based on the four vertices of the page region of the image to be processed.

[0018] A fourth aspect of the present invention also provides an image correction apparatus for identifying and correcting a subject region in an image. The apparatus includes: a calculation module for using four vertices of the image as reference points and employing regression calculation to determine a number of positioning key points, which are used to determine the subject region of the image; and a correction module for identifying and correcting the subject region based on the positioning key points to remove irrelevant information and normalize the subject region.

[0019] A fifth aspect of the present invention also provides a computer device including a processor and a memory, the memory being used to store a computer-executable program, wherein when the computer program is executed by the processor, the processor performs an image correction method as described in any one of the first and second aspects of the present invention.

[0020] A sixth aspect of the present invention also provides a computer program product storing a computer-executable program, which, when executed, implements the image correction method as described in any one of the first and second aspects of the present invention.

[0021] (III) Beneficial Effects

[0022] Compared with existing technologies, this invention obtains the key points for locating the main body region of the image to be processed by using the four vertices of the image as reference points for regression calculation. Based on these key points, the main body region is directly extracted from the image to be processed for correction. This solution can accurately identify the main body region in the image to be processed without image segmentation, thus improving image processing speed and effectively avoiding the problem of unstable image segmentation edges. This solution can also remove irrelevant background outside the main body region.

[0023] Furthermore, by calculating the confidence level of the location coordinates of key points, this method can identify whether there is a main subject area in the image to be processed, so as to determine whether to continue subsequent correction processing.

[0024] Furthermore, in some application scenarios, such as image search, where the main area of ​​the image is the page area, image annotation can more accurately identify the four vertices of the page area in various situations. This allows for the subsequent creation of a training dataset using images annotated with the four vertices of the page area, and the training dataset can then be used to train an image localization regression model. Even further, for example, a computer can be used to infer the positions of page vertices outside the sample image based on the page vertices located within the sample image and the intersections of the page edges and image edges.

[0025] The trained image localization regression model is used to calculate the localization key points and confidence scores of the image to be processed. Based on the calculated localization key points, the page region is directly extracted from the image to be processed. This can more accurately identify the page region in the image to be processed under various conditions (such as tilt), without the need for image segmentation, thus improving the speed of image processing and effectively avoiding the problem of unstable image segmentation edges. At the same time, it can more accurately identify whether there is a page region in the image to be processed, and also improve the image classification accuracy. This solves the problem of low image recognition accuracy caused by factors such as page region tilt relative to the image, presence of irrelevant background, and non-subject areas, and can more accurately identify images with or without page regions. Attached Figure Description

[0026] Figure 1 These are schematic diagrams of the images before and after correction in Embodiment 1 of the present invention;

[0027] Figure 2 This is a flowchart of an example of the image correction method according to Embodiment 1 of the present invention;

[0028] Figure 3 This is a schematic diagram of the image vertices in the page area obtained by regression calculation in the image correction method of Embodiment 1 of the present invention;

[0029] Figure 4 This is a flowchart of another example of the image correction method according to Embodiment 1 of the present invention;

[0030] Figure 5 This demonstrates the application. Figure 4 A schematic diagram of an example image to be processed for the image correction method;

[0031] Figure 6 This is a schematic diagram illustrating the principle of establishing an image localization regression model using a regression algorithm in the image correction method of Embodiment 1 of the present invention;

[0032] Figure 7 This is a flowchart illustrating yet another example of the image correction method according to Embodiment 1 of the present invention;

[0033] Figure 8 This is a schematic diagram illustrating an example of annotation processing of page regions in an image to be processed using the image correction method of Embodiment 1 of the present invention;

[0034] Figure 9 This is a schematic diagram illustrating another example of annotation processing of page regions in an image to be processed using the image correction method of Embodiment 1 of the present invention;

[0035] Figure 10This is a schematic diagram illustrating yet another example of annotation processing of page regions in an image to be processed using the image correction method of Embodiment 1 of the present invention;

[0036] Figure 11 This is a schematic diagram illustrating yet another example of annotation processing of page regions in an image to be processed using the image correction method of Embodiment 1 of the present invention;

[0037] Figure 12 It shows Figure 5 A schematic diagram of the corrected image to be processed;

[0038] Figure 13 This is a flowchart illustrating yet another example of the image correction method according to Embodiment 1 of the present invention;

[0039] Figure 14 This is a flowchart illustrating an example of the method for constructing a page detection model according to Embodiment 2 of the present invention;

[0040] Figure 15 This is a flowchart illustrating an example of the image correction method for the page detection model in Embodiment 3 of the present invention;

[0041] Figure 16 This is a schematic diagram of an example of the image correction device according to Embodiment 4 of the present invention;

[0042] Figure 17 This is a schematic diagram of an example of the image correction device according to Embodiment 4 of the present invention;

[0043] Figure 18 This is a schematic diagram of another example of the image correction device according to Embodiment 4 of the present invention;

[0044] Figure 19 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0045] Figure 20 This is a schematic diagram of a computer program product according to an embodiment of the present invention. Detailed Implementation

[0046] In the description of specific embodiments, detailed descriptions of structures, performance, effects, or other features are provided to enable those skilled in the art to fully understand the embodiments. However, this does not preclude those skilled in the art from implementing the present invention with technical solutions that do not contain the aforementioned structures, performance, effects, or other features under specific circumstances.

[0047] The flowchart in the accompanying drawings is merely an exemplary process demonstration and does not imply that the solution of this invention must include all the content, operations, and steps in the flowchart, nor does it imply that they must be executed in the order shown in the diagram. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. Without departing from the inventive spirit of this invention, the execution order shown in the flowchart can be changed according to the actual situation.

[0048] The box in the attached diagram Figure 1 Generally, these refer to functional entities, and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processing unit devices and / or microcontroller devices.

[0049] The same reference numerals in the accompanying drawings denote the same or similar elements, components, or parts, and therefore, repeated descriptions of the same or similar elements, components, or parts may be omitted below. It should also be understood that although terms such as first, second, third, etc., indicating numbers may be used herein to describe various devices, elements, components, or parts, these devices, elements, components, or parts should not be limited by these terms. That is, these terms are only used to distinguish one from another. For example, a first device may also be referred to as a second device, without departing from the essential technical solution of the invention. Furthermore, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0050] In some image search scenarios, such as searching for questions by taking a picture, the images taken by users usually contain a lot of irrelevant information, such as background areas, incomplete questions, etc. On the other hand, the angle from which users take pictures is usually not perpendicular to the page, which will cause near objects to appear larger and far objects to appear smaller. These factors will cause a lot of interference to subsequent detection and recognition.

[0051] To address these issues, this solution aims to identify and correct the main subject area of ​​the captured image. The correction effect of this solution is as follows: Figure 1 As shown.

[0052] This invention proposes an image correction method. This method uses the four vertices of the image as reference points to perform regression calculations, determining the key positioning points of the main body region. The area enclosed by the lines connecting these key positioning points is defined as the main body region. Based on these key positioning points, the main body region is corrected to remove irrelevant information and standardize the main body region. The main body region can be, for example, a page area in a photo-based question-solving scenario. The page area in the photo may exhibit distortion, missing corners, or other issues requiring correction. Identifying the key positioning points of the page area, such as the four vertices, can be used to correct the page area, for example, by converting it into a more standard rectangular region, resulting in a corrected image.

[0053] Therefore, this method obtains the key points for locating the subject region to be corrected by performing regression calculations using the four vertices of the image as reference points, and directly extracts the subject region from the image to be processed based on these key points. This method can accurately identify the subject region in the image to be processed without image segmentation, thus improving the speed of image processing and effectively avoiding the problem of unstable image segmentation edges. At the same time, it can accurately identify whether there is a subject region in the image to be processed.

[0054] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0055] Figure 2 This is a flowchart of an example of the image correction method according to Embodiment 1 of the present invention.

[0056] like Figure 2 As shown, the image correction method includes the following steps.

[0057] Step S101: Using the four vertices of the image as reference points, regression calculation is used to determine several key positioning points, which are used to determine the main body area of ​​the image.

[0058] This step uses the four vertices of the image as reference points to perform regression calculations, obtaining several key positioning points for determining the main body region. The area enclosed by the lines connecting these key positioning points is the main body region.

[0059] In some specific embodiments, for example, in the scenario of searching for answers by taking a picture, the main area of ​​the image includes the page area, and the several positioning key points include the four vertices of the page area (i.e., the main area).

[0060] Optionally, the four vertices of the image are determined, and a bounding box regression algorithm is used as reference points to perform regression calculations, obtaining four key positioning points in the image. The area enclosed by the lines connecting these four key positioning points is taken as the page area. See [link to details] for further information. Figure 3 . Figure 3 The four darker points on the outer side are the four vertices of the image, while the four lighter points on the inner side are the four key points for locating the main area (i.e., the page) obtained through regression calculation.

[0061] It should be noted that the bounding box regression algorithm essentially uses the four vertices of the image as reference points to perform regression calculations to obtain the key points for locating the main subject region in the image (i.e., the image to be processed). See [link to documentation] for details. Figure 3 The main area can also be Figure 1 The four-sided border area shown in the figure is located at the four vertices of the four-sided border.

[0062] In this embodiment, the positioning key points are used to determine the main body area of ​​the image. The selection and number of positioning key points vary depending on the shape of the main body area, but as long as the area enclosed by lines connecting adjacent positioning key points can determine the main body area, this embodiment does not impose specific limitations. For example, in a photo-based question-searching scenario, the main body area of ​​the image is the page area of ​​a book, exam paper, etc. In this case, the corresponding positioning key points can be selected from the four vertices of the page area. In other embodiments, the midpoint of the page edge can also be added as a positioning key point. Step S102: Based on the positioning key points, the main body area is identified and corrected to remove irrelevant information and standardize the main body area.

[0063] In this step, if the key points for determining the main area of ​​the image have been obtained, irrelevant information in the application scenario can be removed based on these points, and the image distortion can be corrected.

[0064] For example, in scenarios where questions are searched by taking a photo, such as... Figure 1 , Figure 3 and Figure 5 As shown, the content within the box, which is the main area of ​​the image, is what the user wants, but its edges contain a lot of irrelevant information, such as the background area, the afterimage, etc. Step S101 obtains the four vertices of the main area in the image (e.g., Figure 1 The four vertices of the square frame, as follows Figure 3Four points (lighter in color on the inner side) are used as positioning key points. In step S102, the page area can be corrected based on the four positioning key points calculated in step S101. The part outside the page area (or main area) determined by the four vertices is removed, and the page area is converted into a normalized shape, such as a rectangle commonly found on pages, or a long rectangle commonly found in title areas.

[0065] For recognition scenarios involving pages and exam papers, such as image-based question searching, the area outside the page area / frame area defined by the four vertices is considered irrelevant information. In other embodiments, such as license plate recognition, the portion outside the license plate area determined by the location key points is considered irrelevant information.

[0066] More specifically, in some embodiments, if the main area required by the user, such as the page area, is generally text information, then other information unrelated to the text information, such as background information (and in some cases, graphics), can be removed as irrelevant information.

[0067] Furthermore, the shape of the main area in the image, determined by the positioning key points, can be corrected and standardized according to the shape of the main area in different usage scenarios. The specific method for standardizing the main area can be determined by those skilled in the art based on the specific application scenario. For example, when adjusting the shape of a page area in a book, the general dimensions of a book page can be referenced for appropriate processing, including adjusting the tilt angle of the page area and correcting the rotation direction of the image, etc.

[0068] It should be noted that the method of the present invention is particularly suitable for identifying and correcting page regions in images. This method is also particularly suitable for correcting images obtained by the server of an educational service product APP, where the image is the question image used by the user for querying. The following will use the correction of the question image as an example to specifically describe the method of the present invention; however, those skilled in the art should understand that the application of the present invention is not limited thereto.

[0069] Figure 4 This is a flowchart of another example of the image correction method according to Embodiment 1 of the present invention.

[0070] exist Figure 4 In the example shown, the image correction method may further include step S201 of acquiring the image to be processed.

[0071] In step S201, the image to be processed is obtained and then corrected.

[0072] Specifically, the image to be processed may include the page area to be corrected, a background area unrelated to the page area, a captured incomplete title, and / or other unrelated background items. The page area may be tilted at a specific angle relative to the entire display border, and this page area contains the title to be identified.

[0073] Next, in step S202, regression calculation is performed using the four vertices of the image to be processed as reference points to obtain several key positioning points for determining the main body region. The area enclosed by the lines connecting these key positioning points is the page region (i.e., the main body region).

[0074] Furthermore, the page area is identified and corrected to remove irrelevant information such as background areas and photographic residue, and the page area is standardized.

[0075] It should be noted that, due to Figure 4 Steps S202 and S203 in Figure 2 Steps S101 and S102 are largely the same, so specific descriptions of steps S202 and S203 are omitted.

[0076] Figure 5 The diagram illustrates an example of an image to be processed, and shows the application... Figure 4 The image correction method is used for correction processing.

[0077] like Figure 5 As shown, the image to be processed is a picture of a question taken by a user for checking or correcting answers. In this image, the area enclosed by the gray quadrilateral is the main subject area the user actually wants to correct, i.e., the page area to be corrected. This page area is tilted at an angle of less than 30 degrees relative to the entire display border. The image also includes a background object (keyboard) and an area without questions (the correction column area in the figure). Therefore, this embodiment of the invention aims to automatically identify the page area in the image to be processed in order to correct that page area.

[0078] Furthermore, in order to identify the page regions in the image to be processed (i.e., identify the...) Figure 5 The area enclosed by the gray quadrilateral frame is used as a reference point for regression calculation to determine the four key positioning points of the page area.

[0079] In this example, the step of determining the location key points through regression calculation can be implemented using an image localization regression model, which can identify the page regions in the image that need to be corrected. Of course, the image localization regression model needs to be built and trained in advance before using it for calculation.

[0080] Optionally, the image localization regression model described above can be established, and the trained image localization regression model can be used to obtain the location information of the localization key points, such as the coordinates of the localization key points. For example, the image localization regression model described above can be established based on a bounding box regression algorithm.

[0081] It should be noted that the bounding box regression algorithm essentially uses the four vertices of the image as reference points to perform regression calculations to obtain the key points for locating the main body region to be corrected. For example, these could be the four vertices (or four corner points) of the page area. See [link to documentation] for details. Figure 3 and Figure 6 .

[0082] For example, in the original graph space, the information of a box can be represented by four values, namely the four vertices. This indicates (or describes) the assumption that, during model training, the ground truth box (GT box) is used. We define an anchor (a predefined box). (In this invention, the four vertices of the image are selected as anchors, i.e., predefined boxes A), and the ground truth bounding box predicted by the model is the offset of G relative to the predefined box A. , The first vertex is calculated as follows:

[0083] (1)

[0084] (2)

[0085] in, It is the x-coordinate of a vertex. It is the ordinate value of a vertex; w and h are the length and width of the image, respectively; These are the predefined hyperparameters. The formulas for the other three vertices are the same as formulas (1) and (2), only requiring adaptive modification of the corresponding vertex numbers. , , , , , Therefore, the description of the formulas for the other three vertices is omitted.

[0086] If the model's output The corresponding regression loss is as follows.

[0087] (3)

[0088] Furthermore, P and A can be converted into the actual bounding boxes to be predicted.

[0089] (4)

[0090] (5)

[0091] in, It is the x-coordinate value of a key location point in the main area. It is the ordinate value of a vertex; w and h are the length and width of the image, respectively; These are predefined hyperparameters.

[0092] It should be noted that the other three vertices The formula is the same as formulas (4) and (5), only the corresponding vertex numbers need to be adapted. Therefore, the description of the formulas for the other three vertices is omitted.

[0093] Therefore, by using the above formulas (1) to (5), the coordinates of the four vertices of the image can be converted and calculated to obtain the coordinates of the four vertices corresponding to the main body area.

[0094] Optionally, such as Figure 7 As shown, the image correction method of the present invention further includes: establishing an image localization regression model, and using the trained image localization regression model to obtain (or output) the location information of the localization key points of the image to be processed. In other words, step S101 may include the step of establishing an image localization regression model (step S301) and the step of using the image localization regression model to obtain (or output) the location information of the localization key points (step S302), see details below. Figure 7 .

[0095] Specifically, establishing the image localization regression model includes: building an initial image localization regression model; labeling sample data to establish a training dataset; and training the image localization regression model using the sample data in the training dataset. The labeled sample data includes labeling sample images to mark key points used to determine the location of the main subject area and whether the main subject area is included. For example, in a full-page image search application scenario, the four vertices of the page area and whether the page area is included can be labeled.

[0096] More specifically, in some embodiments, data annotation of the sample image includes labeling whether it contains a page region and labeling the four vertices of the page region if it does contain one. However, the four vertices of the page region are not all within the shooting range (i.e., within the sample image). If a vertex is not in the sample image, its position needs to be estimated, and then labeled based on the estimated position to obtain the annotation points for the four location key points, thus obtaining a sample image labeled with the four location key points. The labeled sample image can be used as training sample data to build a training dataset and to train the image localization regression model.

[0097] In this example, the annotation of key points for locating the sample image typically includes the following situations: no page area (i.e., main area) exists in the image; none of the four vertices of the page area are within the sample image; one vertex of the page area to be annotated is not within the sample image; two vertices of the page area to be annotated are not within the sample image; and none of the three vertices of the page area to be annotated are within the sample image.

[0098] Specifically, for sample images where no page area exists, no location key points are marked; instead, the non-existent page area is directly marked. When the page area of ​​a sample image is entirely within the image itself, the four vertices of that page (i.e., the page area in the image) are directly marked as the location key points. See [link to documentation] for details. Figure 8 As shown. When the page area of ​​the sample image extends beyond the sample image, i.e., all four vertices of the page area are outside the sample image, the four vertices of the image are directly marked as the positioning key points. See [link to documentation] for details. Figure 9 As shown.

[0099] When a portion of the page area is located outside the sample image (i.e., at least one of the four vertices of the page area is outside the sample image), the position of the page vertex located outside the sample image is calculated based on the page vertex located inside the sample image and the intersection of the page edge and the edge of the image (which can also be considered as the intersection of the page area and the boundary line of the image, hereinafter referred to as the intersection point). The page vertex inside the image and the calculated page vertex outside the image are then marked together as the positioning key points of the page area.

[0100] Specifically, for cases where a portion of the page area is located outside the sample image, the steps for marking and locating key points include the following sub-steps:

[0101] Step 1: First, mark the vertices of the sample image located within the image, and the intersection points of the page edge and the image edge.

[0102] This step can generally be done manually. In other embodiments, this step can also be achieved by automatically labeling vertices and their intersections by recognizing the edges of the page and the image.

[0103] For example, if a vertex of the page area to be labeled is within the sample image, four or five points need to be labeled in the sample image first (the number of vertices in the page area within the image plus the number of intersection points). Then, the position of the vertex outside the sample image can be deduced based on this. See [link to documentation] for details. Figure 10 As shown.

[0104] Figure 10This is another example of the annotation processing (five annotation points) of the page area of ​​the image used as a sample image.

[0105] like Figure 10 As shown, one vertex in the lower right corner of the page area in the sample image is not in the image. Figure 10 Five points (a, b, c, d, e) need to be marked in the sample image, represented by gray dots. Among them, three marked points (a, b, c) are vertices of the page area within the sample image, and two marked points (d, e) are the intersections of the page edge and the image edge (i.e., the image boundary line).

[0106] like Figure 11 As shown, the bottom two vertices of the page area in the sample image are not present in the image. For Figure 11 The sample image needs to be marked with six points. Two of these points are the vertices of the page area (the two points at the top, represented by a gray outer circle and gray dots), and the other four points are the intersections of the page area and the image edge, represented by a gray outer circle and black dots.

[0107] There may also be cases where none of the three vertices of the page area are in the image, but only one vertex is in the image. The situation of labeling vertices and intersections is similar to that above, and will not be described in detail here.

[0108] After marking the vertices and intersections of the page area within the image (hereinafter collectively referred to as marked points) in the sample image, the vertex positions of the page area outside the image are calculated using the following method.

[0109] Step 2: Traverse the marked points (i.e., the vertices and intersections marked by the method described above; the computer cannot distinguish between vertices and intersections when processing), and determine whether a marked point is an intersection. The method for determining an intersection includes: determining whether two adjacent marked points are simultaneously located on the same image edge; if two adjacent marked points are simultaneously located on the same image edge, then further determine whether the angle formed by the intersection of the extended edges of the two page areas where these two marked points are located satisfies a first threshold angle (preferably 60 degrees); if the angle satisfies a first threshold angle, then the two adjacent marked points are determined to be intersections, and the two adjacent marked points are marked as non-vertices.

[0110] Step 3: The edges of the two page areas corresponding to (or containing) the two annotation points (e.g., ...) Figure 10 The intersection point formed by extending and intersecting the ae and cd edges in the image is used as the predicted vertex of the page region located outside the image, and the position of this predicted vertex is labeled (e.g., ...). Figure 10 The predicted vertices of the middle page area are labeled as f), and the two labeled points that were determined to be intersections are deleted (e.g., Figure 10 (d and e in the text).

[0111] After traversing all the marked points according to the above steps, the remaining marked points and the predicted vertices located outside the image are used together as the positioning key points of the page area (four positioning key points in this example) and output to the subsequent correction steps. In step S102, the page area is processed by cutting, distortion correction and other processes based on these positioning key points.

[0112] If two adjacent annotation points are not located on the same image edge, the process continues to iterate through the subsequent annotation points.

[0113] Specifically, if two adjacent annotation points are located on the same image edge (both points are at the image edge), but the angle formed by the intersection of the extended edges of the two page regions containing these two annotation points is less than a first angle threshold (preferably 60 degrees), then these two annotation points are considered normal vertices, i.e., as shown below. Figure 8 The two top markers shown are considered normal vertices because the angle formed by the intersection of the extended edges of the two page areas containing these markers is less than 60 degrees.

[0114] When the included angle is not less than the first angle threshold, it indicates that the two labeled points are not vertices and are redundant intersections that need to be deleted. Simultaneously, vertices outside the image are predicted by extending the edges of adjacent page regions to intersect. The preferred first angle threshold is 60 degrees. This value was obtained through experimental statistics; 99% of images meeting this 60-degree condition have correct labeling and can be used to train the model.

[0115] In a specific example, for Figure 10 As shown, the vertex positions of the page area outside the image are calculated using the method described above. Traversal... Figure 10 In the example, for points a, b, c, d, and e, we sequentially determine whether two adjacent annotation points are located at the intersection of the same image edge. When determining whether two adjacent annotation points (e.g., d and e) are located at the intersection of the same image edge, we determine whether the included angle β formed by the intersection of the two adjacent page edges (the edge ae where a and e are located and the edge cd where c and d are located) of the page area where the two intersection points are located is greater than or equal to 60 degrees.

[0116] Furthermore, when the included angle β is greater than or equal to 60 degrees, the two adjacent annotation points are determined to be intersection points (redundant points to be deleted). The intersection point (or point of intersection) formed by extending the two adjacent page edges (ae and cd) of the page area where the two intersection points (d and e) are located is taken as the predicted vertex position (or vertex for position estimation) of the page area. The predicted vertex is labeled as f, and the two annotation points d and e are deleted at the same time to obtain a sample image with four annotation points (a, b, c, f), which is used as training sample data.

[0117] In another case, for example, when determining two adjacent labeled points (e.g. Figure 10 If a, ae, bc, and cd are not intersections of the same image edge, then at least one of these annotation points is a vertex of the page area.

[0118] In another case, when determining two adjacent marker points (e.g.) Figure 10 When d and e) are the intersection points of the same image edge, but the two adjacent page edges of the page area where these two intersection points are located (e.g., Figure 10 If the angle β formed by the intersection of the extended sides ae (containing a and e) and cd (containing c and d) is less than 60 degrees, then the two adjacent labeled points are normal vertices in the image.

[0119] It should be noted that the 60 degrees mentioned above is determined by those skilled in the art based on the actual situation. For example, the first threshold may be determined based on requirements such as accuracy or model evaluation parameters, to ensure that the identification of images containing page areas (or subject areas) is reliable.

[0120] Figure 11 This is a schematic diagram illustrating yet another example of annotation processing (six annotation points) of a page area in an image to be processed using the image correction method of Embodiment 1 of the present invention.

[0121] like Figure 11 As shown, the sample image has six annotation points, represented by white dots with a black outer ring and black dots. Two of these annotation points are vertices of the page area, represented by white dots with a black outer ring (a, b); four annotation points (c, f, d, e) are the intersections of the page area and the image edge, represented by black dots; and predicted vertices are represented by gray dots with a black outer ring.

[0122] for Figure 11 In the case shown, the use of Figure 10The same method is used to determine whether two adjacent annotation points are located at the intersection of the same image edge. After the determination, annotation point cd is considered to be at the intersection of the same image edge, and annotation point ef is considered to be at the intersection of the same image edge.

[0123] from Figure 11 As can be seen from this, the two labeled points cd are located at... Figure 11 On the left edge of the first image, the other two labeled points ef are located on... Figure 11 The second image edge on the right side, opposite to the edge of the first image, in this case, determines whether the first included angle β1 formed by the intersection of the extension of the page edge bc and the page edge de is greater than or equal to 60 degrees, and determines whether the second included angle β2 formed by the intersection of the extension of the page edge af and the page edge de is greater than or equal to 60 degrees.

[0124] Specifically, when both the first included angle β1 and the second included angle β2 are greater than or equal to 60 degrees, the vertices of the first included angle and the second included angle are labeled g and h respectively, and the labeled points g and h are used as the predicted vertices (or vertices for position estimation) of the page area. The corresponding labeled points c, d, e, and f are deleted to obtain a sample image labeled with four labeled points a, b, g, and h, which is used as training sample data.

[0125] In addition, there are cases where three vertices of the page area are not within the image boundary. In this case, the number of annotation points is seven or eight, in order to match... Figure 10 , Figure 11 The same method is used to estimate the position of vertices in the page area and remove non-vertices (or the intersection of the page area edge and the image edge) to obtain sample images of four labeled points, which are then used as training sample data.

[0126] Optionally, the vertices of the page region located outside the sample image can be automatically obtained by a computer program using the method described above. That is, for images in the sample image that are missing vertices, the positions of the vertices located outside the sample image can be calculated / inferred to achieve the annotation of the image, and the training dataset can be established using the annotated image samples.

[0127] Optionally, data annotation of sample images may further include: labeling the images to be processed as title images and non-title images to obtain title image labels and non-title image labels, thereby obtaining sample data for classifying title images and non-title images. The label values ​​(reliable values ​​in this example) for both title image labels and non-title image labels are values ​​between 0 and 1, and a predetermined label threshold is set. When the label value of the image to be processed is greater than the predetermined label threshold, the image to be processed is determined to be a title image; when the label value of the image to be processed is less than the predetermined label threshold, the image to be processed is determined to be a non-title image.

[0128] Therefore, the image localization regression model is trained using all the labeled training sample data (the labeled images). The trained image localization regression model is then used as input to output the coordinates and confidence scores of the localization keypoints in the image. For images whose main area is a page, the image localization regression model outputs the coordinates of the four vertices (i.e., localization keypoints) of the page area.

[0129] Specifically, the coordinates of the location key points can be calculated using a trained image localization regression model. The image to be processed is input into the image localization regression model, which outputs the position coordinates and confidence scores of four location key points in the page area of ​​the image. These four location key points are the coordinates of the four vertices of the page area (x1, y1, x2, y2, x3, y3, x4, y4), and the confidence score represents the reliable probability of the output location key points.

[0130] It should be noted that when the confidence level of the image to be processed is less than a predetermined threshold, it indicates that the input image does not contain a main subject region (e.g., there is no page area in the image to be processed) or the confidence level of the main subject region is very low. In this case, there is no need for subsequent image correction. Therefore, when the confidence level of the location coordinates of the key points is less than the predetermined threshold, the correction of the main subject region is stopped and a warning message indicating no main subject region is output.

[0131] In summary, by using the sample data processed with the above annotations to establish a training dataset, the four vertices of the page region in various situations can be more accurately annotated. Using this training dataset to train an image localization regression model, and then using the trained model to calculate the four localization key points and confidence scores of the page region (i.e., the page region to be corrected) in the image to be processed, the page region within the image to be processed can be identified more accurately. Simultaneously, it can more accurately identify whether a page region exists in the image to be processed. This solves the problem of low image recognition accuracy caused by factors such as the page region being tilted relative to the image, having irrelevant backgrounds, or being a non-subject area. It can also accurately identify images with or without page regions. Next, in step S102, based on the localization key points, the subject region is identified and corrected to remove irrelevant information and normalize the subject region.

[0132] Specifically, based on the four calculated positioning key points, the page area to be corrected is directly extracted from the image to be processed without image segmentation, thereby improving the speed of image processing.

[0133] Furthermore, the page area to be corrected is corrected by adjusting the four positioning key points (e.g., by adjusting the positioning key points). Figure 5 The four key positioning points (abcd) of the middle page area are transformed and adjusted to the four vertices of the image to be processed. Figure 5 The four vertices ABCD of the image are used to convert the page area into a rectangular area (i.e., to a rectangular area). Figure 5 The image contains four vertices (ABCD). The transformation adjustments include stretching, scaling, and distortion correction of the image.

[0134] Specifically, the correction may include radial transformation and / or orientation correction of the page area.

[0135] Figure 12 It shows Figure 5 A schematic diagram of the corrected page area (i.e., the main area) of the image to be processed.

[0136] like Figure 12 As shown, after the above correction process, the four positioning key points of the page area to be corrected (i.e. Figure 5 The four key positioning points (abcd) of the middle page area are transformed and adjusted to the four vertices of the image to be processed (i.e., Figure 5 The four vertices ABCD of the image are used to convert the page area into a rectangular area, removing irrelevant background areas (e.g.) Figure 5 (Irrelevant areas such as the keyboard background area and correction column) will also have their orientation adjusted to be positive, conforming to reading habits.

[0137] Optionally, after performing radiometric transformation and / or orientation correction, image font distortion correction and / or binarization processing can be performed to eliminate image noise interference and obtain a corrected image.

[0138] Specifically, the binarization process involves using an adaptive binarization algorithm to binarize the page region to be corrected.

[0139] Furthermore, adaptive binarization algorithms include Otsu binarization algorithm (OSTU binarization algorithm), local thresholding algorithm (Niblack binarization algorithm), etc.

[0140] Therefore, this invention directly extracts page regions from the image to be processed based on four key positioning points, eliminating the need for image segmentation, thus improving image processing speed and effectively avoiding the problem of unstable image segmentation edges. Figure 13 This is a flowchart of yet another example of the image correction method according to Embodiment 1 of the present invention.

[0141] like Figure 13 As shown, the method of the present invention further includes a step S401 of orientation correction of the image to be processed before image correction. Orientation correction here includes transforming the image into a forward orientation that conforms to reading habits through operations such as rotation.

[0142] In the above embodiment, we assume that the image is upright. For images that are not upright, we first adjust the orientation of the current image to be upright, and then correct the page area to be corrected according to the four positioning key points to obtain the corrected image.

[0143] In some embodiments, the image to be processed (which may be upright or not) can first be input into the orientation correction model to obtain an upright image to be processed, and then input into the image localization regression model above to obtain localization key points, and then the main body region can be identified and corrected.

[0144] It should be noted that the above is only an example and should not be construed as a limitation of the present invention.

[0145] The above-described question-searching method is for illustrative purposes only, and there are no particular limitations on the order or number of steps. Furthermore, a step in the above method can be broken down into two or three steps, or some steps can be combined into one step, depending on the specific example.

[0146] Compared with existing technologies, this invention obtains the key points for locating the main body region of the image to be processed by using the four vertices of the image as reference points for regression calculation. Based on these key points, the main body region is directly extracted from the image to be processed for correction. This solution can accurately identify the main body region in the image to be processed without image segmentation, thus improving image processing speed and effectively avoiding the problem of unstable image segmentation edges. This solution can also remove irrelevant background outside the main body region.

[0147] Furthermore, by calculating the confidence level of the location coordinates of key points, this method can identify whether there is a main subject area in the image to be processed, so as to determine whether to continue subsequent correction processing.

[0148] Furthermore, in some application scenarios, such as image search, where the main area of ​​the image is the page area, image annotation can more accurately identify the four vertices of the page area in various situations. This allows for the subsequent creation of a training dataset using images annotated with the four vertices of the page area, and the training dataset can then be used to train an image localization regression model. Even further, for example, a computer can be used to infer the positions of page vertices outside the sample image based on the page vertices located within the sample image and the intersections of the page edges and image edges.

[0149] The trained image localization regression model is used to calculate the localization key points and confidence scores of the image to be processed. Based on the calculated localization key points, the page region is directly extracted from the image to be processed. This can more accurately identify the page region in the image to be processed under various conditions (such as tilt), without the need for image segmentation, thus improving the speed of image processing and effectively avoiding the problem of unstable image segmentation edges. At the same time, it can more accurately identify whether there is a page region in the image to be processed, and also improve the image classification accuracy. This solves the problem of low image recognition accuracy caused by factors such as page region tilt relative to the image, presence of irrelevant background, and non-subject areas, and can more accurately identify images with or without page regions.

[0150] Example 2

[0151] According to a second aspect of the present invention, the present invention also provides a method for constructing a page detection model, the page detection model being used to detect the four vertices of a page region of an image.

[0152] Figure 14 This is a schematic diagram illustrating an example of a method for constructing a page detection model according to Embodiment 2 of the present invention.

[0153] like Figure 14 As shown, the method includes the following steps:

[0154] S501, build a key point detection model.

[0155] S502, using an image labeled with the four vertices of the page region, the keypoint detection model is trained so that the trained keypoint detection model can output the four vertices of the page region and their confidence scores.

[0156] Specifically, the image with the four vertices of the page area marked is completed through the steps of annotating the sample image described in Example 1.

[0157] Furthermore, the image to be processed is input into the key point detection model, which outputs the position coordinates and confidence scores of four key points corresponding to the image. These four key points are the coordinates of four vertices of the page area (x1, y1, x2, y2, x3, y3, x4, y4), and the confidence score represents the reliable probability of the output key points.

[0158] It should be noted that in Example 2, the descriptions of the parts that are the same as in Example 1 have been omitted.

[0159] Compared with existing technologies, the present invention detects the four vertices of the page region of an image using a page detection model, which can more accurately detect the four vertices of the page region of the image, that is, can more accurately determine the page region to be corrected.

[0160] Example 3

[0161] According to a third aspect of the present invention, the present invention also provides an image correction method.

[0162] like Figure 15 As shown, the method includes the following steps:

[0163] Step S601: Input the image to be processed into the page detection model generated according to the model building method described in Example 2, and obtain the four vertices and confidence scores of the page region of the image to be processed.

[0164] Step S602: Determine whether the confidence level is greater than or equal to the first threshold. When the confidence level is greater than the first threshold, perform page region identification and correction on the image to be processed based on the four vertices of the page region of the image to be processed.

[0165] It should be noted that the specific value of the first threshold can be determined by those skilled in the art based on the actual situation, such as the requirements of accuracy or model evaluation parameters, so as to ensure that the identification of images containing page areas (or correction areas) is reliable.

[0166] For example, if the first threshold is 0.75, and the confidence level calculated in step 601 is greater than or equal to 0.75, then the page area (i.e., the area to be corrected) is further determined based on the four vertices (or positioning key points) of the image to be processed, and the main body area is cropped and corrected. If it is less than 0.75, it indicates that the confidence level of the positioning key points used to determine the main body area is too low, meaning that the current image does not have a main body area or the coordinates of the positioning key points are unreliable. In this case, it is best to check the shooting situation of the image, and it is not necessary to use the method described in Example 1 to correct the area to be corrected.

[0167] It should be noted that in Example 3, the descriptions of the parts that are the same as those in Example 1 and Example 2 are omitted.

[0168] Compared with existing technologies, the present invention detects the four vertices of the page region of an image through a page detection model, which can more accurately detect the four vertices of the page region of the image, that is, can more accurately determine the page region to be corrected (i.e., the region to be corrected), without the need for image segmentation processing, thus improving the speed of image processing, effectively avoiding the problem of unstable image segmentation edges, and can accurately identify whether there is a page region in the image to be processed.

[0169] Example 4

[0170] The following describes an embodiment of the apparatus of the present invention, which can be used to perform the method embodiments of the present invention. The details described in the apparatus embodiments of the present invention should be considered as supplements to the above method embodiments; details not disclosed in the apparatus embodiments of the present invention can be implemented with reference to the above method embodiments.

[0171] Reference Figures 16 to 18 The following will describe the image correction device 100 of Embodiment 2 of the present invention.

[0172] According to a fourth aspect of the present invention, the present invention also provides an image correction device 100 for identifying page regions in an image and correcting the page regions. The image correction device 100 includes: an acquisition module 110 for acquiring an image to be processed; a calculation module 120 for using the four vertices of the image as reference points and employing regression calculation to determine a number of positioning key points, which are used to determine the main body region of the image; and a correction module 130 for identifying and correcting the main body region based on the positioning key points, so as to remove irrelevant information and normalize the main body region.

[0173] It should be noted that in other application examples, the image correction device 100 of the present invention may not include the acquisition module, but only the calculation module 120 and the correction module 130. See details below. Figure 17 .

[0174] Optionally, the main area of ​​the image includes the page area, and the plurality of positioning key points include the four vertices of the page.

[0175] like Figure 18 As shown, the image correction device 100 of the present invention further includes a model building module 111, which is used to build an image localization regression model and calculate the coordinates of the localization key points using the trained image localization regression model.

[0176] Specifically, the sample data is labeled and used to train the image localization regression model. The labeled sample data refers to labeling the sample image to mark the four key points of the page area in the sample image and whether the page area is included.

[0177] Furthermore, obtaining (or outputting) the coordinates of the location key points using the trained image localization regression model includes: inputting the image to be processed into the image localization regression model, and outputting the position coordinates and confidence scores of four location key points in the page area of ​​the image to be processed. The confidence score represents the reliable probability of the output location key points.

[0178] Specifically, it also includes a labeling processing module 112, which is used to label the sample image. When the entire page area of ​​the sample image is within the sample image, the four vertices of the page are directly labeled as the positioning key points. When the page area of ​​the sample image is partially outside the sample image, the positions of the page vertices outside the sample image are calculated based on the page vertices within the sample image and the intersection of the page edge and the image edge. The page vertices within the image and the calculated page vertices outside the sample image are then labeled as the positioning key points.

[0179] Optionally, the step of calculating the position of page vertices located outside the sample image includes: marking the vertices of the sample image located within the image, and the intersection of the page edge and the image edge; traversing all marked points of the sample image, and sequentially determining whether two adjacent marked points are simultaneously located on the same image edge, wherein the marked points include the vertices located within the image and the intersection points; when it is determined that the two adjacent marked points are simultaneously located on the same image edge, further determining whether the included angle formed by the intersection of the extended edges of the two page regions where the two marked points are located satisfies a first threshold angle; when the included angle satisfies a first threshold angle, marking the two adjacent marked points as non-vertices, and marking the intersection formed by the extended edges of the two page regions corresponding to the two non-vertices as the predicted vertex of the page region located outside the image, while deleting the two marked non-vertices, and finally obtaining a sample image marked with four positioning key points.

[0180] Optionally, the first angle threshold is 60 degrees.

[0181] Furthermore, the correction of the page area includes: performing radial transformation and / or orientation correction on the page area based on the four positioning key points to obtain a corrected image.

[0182] It should be noted that in Example 4, the descriptions of the parts that are the same as those in Examples 1, 2 and 3 are omitted.

[0183] Compared with existing technologies, this invention obtains the key points for locating the main body region of the image to be processed by using the four vertices of the image as reference points for regression calculation. Based on these key points, the main body region is directly extracted from the image to be processed for correction. This solution can accurately identify the main body region in the image to be processed without image segmentation, thus improving image processing speed and effectively avoiding the problem of unstable image segmentation edges. This solution can also remove irrelevant background outside the main body region.

[0184] Furthermore, by calculating the confidence level of the location coordinates of key points, this method can identify whether there is a main subject area in the image to be processed, so as to determine whether to continue subsequent correction processing.

[0185] Furthermore, in some application scenarios, such as image search, where the main area of ​​the image is the page area, image annotation can more accurately identify the four vertices of the page area in various situations. This allows for the subsequent creation of a training dataset using images annotated with the four vertices of the page area, and the training dataset can then be used to train an image localization regression model. Even further, for example, a computer can be used to infer the positions of page vertices outside the sample image based on the page vertices located within the sample image and the intersections of the page edges and image edges.

[0186] The trained image localization regression model is used to calculate the localization key points and confidence scores of the image to be processed. Based on the calculated localization key points, the page region is directly extracted from the image to be processed. This can more accurately identify the page region in the image to be processed under various conditions (such as tilt), without the need for image segmentation, thus improving the speed of image processing and effectively avoiding the problem of unstable image segmentation edges. At the same time, it can more accurately identify whether there is a page region in the image to be processed, and also improve the image classification accuracy. This solves the problem of low image recognition accuracy caused by factors such as page region tilt relative to the image, presence of irrelevant background, and non-subject areas, and can more accurately identify images with or without page regions.

[0187] Example 5

[0188] The following describes embodiments of the computer device of the present invention, which can be considered as specific implementations of the methods and apparatus embodiments of the present invention described above. Details described in the computer device embodiments of the present invention should be considered as supplements to the methods or apparatus embodiments described above; details not disclosed in the computer device embodiments of the present invention can be implemented with reference to the methods or apparatus embodiments described above.

[0189] Figure 19 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. The computer device includes a processor and a memory. The memory is used to store a computer executable program. When the computer program is executed by the processor, the processor executes any of the methods in the above embodiments.

[0190] like Figure 19 As shown, the computer device is represented in the form of a general-purpose computing device. There can be one or more processors working collaboratively. This invention also does not exclude distributed processing, meaning that processors can be distributed across different physical devices. The computer device of this invention is not limited to a single entity, but can also be the sum of multiple physical devices.

[0191] The memory stores a computer-executable program, typically machine-readable code. The computer-readable program can be executed by the processor to enable the computer device to perform the method of the present invention, or at least some steps of the method.

[0192] The memory includes volatile memory, such as random access memory (RAM) and / or cache memory, and may also be non-volatile memory, such as read-only memory (ROM).

[0193] Optionally, in this embodiment, the computer device further includes an I / O interface for exchanging data with external devices. The I / O interface can represent one or more of several bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0194] It should be understood that Figure 19 The computer device shown is merely an example of the present invention, and the computer device of the present invention may also include elements or components not shown in the above examples. For example, some computer devices also include display units such as screens, and some computer devices also include human-computer interaction elements such as buttons and keyboards. Any computer device capable of executing a computer-readable program in memory to implement the method of the present invention or at least some steps of the method can be considered as a computer device covered by the present invention.

[0195] Figure 20 This is a schematic diagram of a computer program product according to an embodiment of the present invention. Figure 20 As shown, a computer program product stores a computer-executable program, which, when executed, implements the method described above. The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0196] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0197] From the above description of the embodiments, those skilled in the art will readily understand that the present invention can be implemented by hardware capable of executing specific computer programs, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. included in the system. The present invention can also be implemented by computer software executing the methods of the present invention, for example, by control software executed by a microprocessor, electronic control unit, client, server, etc. However, it should be noted that the computer software executing the methods of the present invention is not limited to execution in one or a specific set of hardware entities; it can also be implemented in a distributed manner using unspecified hardware. For computer software, the software product can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or distributed across a network, as long as it enables computer devices to execute the methods according to the present invention.

[0198] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or computer equipment, and various general-purpose devices can also implement the present invention. The above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image correction method, characterized in that, include: Using the four vertices of an image as reference points, regression calculations are employed to determine several key positioning points. This includes: using a trained image localization regression model to obtain the location information of the key positioning points in the image; wherein, the image is input into the image localization regression model, which uses the four vertices of the image as reference points to perform regression calculations and outputs the location coordinates and confidence scores of the key positioning points in the main body region of the image, where the confidence score represents the reliability probability of the output key positioning point coordinates; and connecting adjacent key positioning points to form the main body region of the image, which includes the page area, and the key positioning points include the four vertices of the page. The establishment and training of the image localization regression model includes: building the initial image localization regression model, labeling sample data to establish a training dataset, and training the initial image localization regression model using the sample data; including: The ground truth box (GT) predicted by the computational model is Compared to predefined boxes offset , The first vertex is calculated as follows: in, It is the x-coordinate of a vertex. It is the ordinate value of a vertex; w and h are the length and width of the image, respectively; For predefined hyperparameters; In the model output The corresponding regression loss values ​​are as follows; The annotation of sample data includes data annotation on whether the sample image contains a page area and annotation of the four vertices of the page area that is contained, in order to indicate whether the sample image is a title image or a non-title image, or to annotate the key points for locating the main body area in the sample image and whether it contains the main body area. When the main body area of ​​the sample image is a page area, the data annotation of the sample image specifically includes: when the page area of ​​the sample image is entirely within the sample image, directly annotating the four vertices of the page as key points for locating the main body area; when the page area of ​​the sample image is partially outside the sample image, the positions of the vertices of the page outside the sample image are inferred based on the vertices of the page within the sample image and the intersection of the page edge and the edge of the sample image, including: annotating the vertices of the sample image that are within the image, and the intersection of the page edge and the edge of the sample image; Iterate through all the labeled points in the sample image and determine whether two adjacent labeled points are simultaneously located on the same edge of the sample image. The labeled points include vertices and intersections within the sample image. When it is determined that two adjacent annotation points are simultaneously located on the edge of the same sample image, it is further determined whether the included angle formed by the intersection of the extended edge lines of the two page regions where the two adjacent annotation points are located satisfies the first threshold angle. When the included angle is greater than or equal to the first threshold angle, two adjacent annotation points are marked as non-vertices, and the intersection point formed by the extension of the two page area edges corresponding to the two non-vertices is marked as the predicted vertex of the page area outside the image. At the same time, the two annotation points marked as non-vertices are deleted, and finally a sample image with four localization key points is obtained. Based on the location key points, the main body region is identified and its orientation is corrected to remove irrelevant information and standardize the main body region.

2. The method according to claim 1, characterized in that, Also includes: When the confidence level is less than the predetermined threshold, the correction of the main body region is stopped and a warning message indicating that there is no main body region is output.

3. The method according to claim 1, characterized in that, Also includes: The first threshold angle is 60 degrees.

4. The method according to claim 1, characterized in that, After identifying the main body region and performing orientation correction, the method further includes: combining image font distortion correction and / or binarization processing to eliminate image noise interference.

5. A method for constructing a page detection model based on the four vertices of a page region in an image, characterized in that, include: Build a key point detection model; The keypoint detection model is trained using an image labeled with the four vertices of the page region, so that the trained keypoint detection model can output the four vertices of the page region and the confidence level. The step of annotating the sample image by means of the method described in any one of claims 1-4, wherein the image with the four vertices of the page area is labeled is completed.

6. An image correction method, characterized in that, include: The image to be processed is input into the page detection model generated by the page detection model construction method according to claim 5, and the four vertices and confidence of the page region of the image to be processed are obtained. Determine whether the confidence level is greater than or equal to a first threshold. When the confidence level is greater than the first threshold, perform page region identification and correction on the image to be processed based on the four vertices of the page region of the image to be processed.

7. An image correction device, characterized in that, include: The calculation module is used to determine several location key points by using the four vertices of an image as reference points and performing regression calculations. This includes: obtaining the location information of the location key points in the image using a trained image localization regression model; wherein, the image is input into the image localization regression model, which uses the four vertices of the image as reference points to perform regression calculations and outputs the location coordinates and confidence scores of the location key points in the main body region of the image, where the confidence score represents the reliability probability of the output location key point coordinates; and, connecting adjacent location key points to form the main body region of the image, which includes the page area, and the location key points include the four vertices of the page. The establishment and training of the image localization regression model includes: building an initial image localization regression model, labeling sample data to establish a training dataset, and training the initial image localization regression model using the sample data. The ground truth box (GT) predicted by the computational model is Compared to predefined boxes offset , The first vertex is calculated as follows: in, It is the x-coordinate of a vertex. It is the ordinate value of a vertex; w and h are the length and width of the image, respectively; For predefined hyperparameters; In the model output The corresponding regression loss values ​​are as follows; Annotating sample data refers to labeling sample images to determine whether they contain page areas and to label the four vertices of the contained page areas. This is done to indicate whether the sample image is a title image or a non-title image, or to label the key points for locating the main body area in the sample image and whether it contains the main body area. Specifically, when the main body area of ​​the sample image is a page area, the data annotation includes: when all page areas of the sample image are within the sample image, directly labeling the four vertices of the page as key points; when part of the page area of ​​the sample image is outside the sample image, calculating the position of the vertices of the page outside the sample image based on the vertices of the page within the sample image and the intersection of the page's edge and the sample image's edge, including: labeling the vertices of the sample image within the image, and the intersection of the page's edge and the sample image's edge. Iterate through all the labeled points in the sample image and determine whether two adjacent labeled points are simultaneously located on the same edge of the sample image. The labeled points include vertices and intersections within the sample image. When it is determined that two adjacent annotation points are simultaneously located on the edge of the same sample image, it is further determined whether the included angle formed by the intersection of the extended edge lines of the two page regions where the two adjacent annotation points are located satisfies the first threshold angle. When the included angle is greater than or equal to the first threshold angle, two adjacent annotation points are marked as non-vertices, and the intersection point formed by the extension of the two page area edges corresponding to the two non-vertices is marked as the predicted vertex of the page area outside the image. At the same time, the two annotation points marked as non-vertices are deleted, and finally a sample image with four localization key points is obtained. The correction module is used to identify the main body region based on the positioning key points, perform orientation correction, remove irrelevant information, and standardize the main body region.

8. A computer device comprising a processor and a memory, the memory being used to store a computer-executable program, characterized in that: When the computer executable program is executed by the processor, the processor performs the image correction method as described in any one of claims 1-4 and 6.

9. A computer program product storing a computer-executable program, characterized in that, When the computer-executable program is executed, it implements the image correction method as described in any one of claims 1-4 and 6.

Images