Corner point detection method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a corner detection method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of image correction. The method comprises the steps of extracting an outer contour of a target object based on an image of the target object; calculating a convex hull based on the points in the outer contour to obtain a convex hull point sequence; determining a target convex quadrangle based on the convex hull point sequence; and obtaining corner coordinates of the target object based on the target convex quadrangle. According to the embodiment of the invention, the operand can be effectively reduced, the target quadrangle can be determined more effectively and quickly, and the target quadrangle can be used for subsequent operation to determine the angular point coordinates of the target object, so that the operation efficiency can be further improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, specifically to the field of image correction technology, and in particular to a corner point detection method, device, electronic device, storage medium and program product. Background Art

[0002] With the popularity of smart devices, document digitization is particularly important. The process of document digitization mainly involves extracting the document area in the captured image and correcting it. Among the related technologies, document digitization is mainly based on deep learning edge detection, but this technology is greatly affected by factors such as complex environments, and it is difficult to improve its accuracy. Summary of the invention

[0003] The embodiments of the present disclosure provide a corner point detection method, device, electronic device, computer-readable storage medium, and computer program product.

[0004] In a first aspect, an embodiment of the present disclosure proposes a corner point detection method, including: extracting the outer contour of a target object based on an image of the target object; calculating a convex hull based on points in the outer contour to obtain a convex hull point sequence; determining a target convex quadrilateral based on the convex hull point sequence; and obtaining the corner point coordinates of the target object based on the target convex quadrilateral.

[0005] In the second aspect, an embodiment of the present disclosure proposes a corner point detection device, including: an outer contour extraction module, configured to extract the outer contour of a target object based on an image of the target object; a convex hull point sequence calculation module, configured to calculate the convex hull based on the points in the outer contour to obtain a convex hull point sequence; a target quadrilateral determination module, configured to determine a target convex quadrilateral based on the convex hull point sequence; and a corner point coordinate generation module, configured to obtain the corner point coordinates of the target object based on the target convex quadrilateral.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the corner detection method described in any implementation method in the first aspect when executing the instructions.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement a corner point detection method as described in any implementation method of the first aspect when executed.

[0008] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the corner point detection method described in any implementation manner in the first aspect.

[0009] In the embodiment of the present disclosure, in the process of processing the image of the target object, a convex hull sequence is obtained based on the outer contour of the target object, and four points that can form a quadrilateral are found based on the points in the convex hull sequence. Since the points on the convex hull are arranged in a certain order (for example, counterclockwise or clockwise), the search process can be effectively simplified and the amount of calculation can be reduced. Moreover, the process of constructing a quadrilateral based on the convex hull can more effectively and quickly determine the target quadrilateral, and use it for subsequent calculations to determine the coordinates of the corner points of the target object, which can further improve the calculation efficiency.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0012] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0013] Figure 2 A flowchart of a corner point detection method provided by an embodiment of the present disclosure;

[0014] Figure 3A A flowchart for determining a target convex quadrilateral based on a convex hull point sequence provided in an embodiment of the present disclosure;

[0015] Figure 3B A flowchart of traversing any combination of four points in a convex hull point sequence to construct a candidate quadrilateral provided in an embodiment of the present disclosure;

[0016] Figure 4 A flowchart of extracting the outer contour of a target object based on an image of the target object provided in an embodiment of the present disclosure;

[0017] Figure 5 A flowchart of another corner point detection method provided by an embodiment of the present disclosure;

[0018] 6A-6C are schematic diagrams comparing the related technology provided by the embodiment of the present disclosure with the embodiment of the present disclosure;

[0019] Figure 7A-7B A schematic diagram of a process of corner point detection in an application scenario provided by an embodiment of the present disclosure;

[0020] Figure 8A structural block diagram of a corner point detection device provided in an embodiment of the present disclosure;

[0021] Fig. 9 A schematic diagram of the structure of an electronic device suitable for executing a corner point detection method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0023] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0024] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the corner detection method, apparatus, electronic device, and computer-readable storage medium of the present disclosure can be applied.

[0025] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0026] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 and server 105 may be installed with various applications for realizing information communication between the two, such as instant messaging applications.

[0027] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0028] The server 105 can provide various services through various built-in applications, such as document-based image processing applications. It should be noted that the original images and other information targeted by the image processing applications can be obtained from the terminal devices 101, 102, 103 through the network 104, or can be pre-stored locally on the server 105 in various ways. Therefore, when the server 105 detects that the data has been stored locally, it can choose to directly obtain the data locally. In this case, the exemplary system architecture 100 may also not include the terminal devices 101, 102, 103 and the network 104.

[0029] Since image processing based on the original image may require more computing resources and stronger computing power, the corner point detection method provided in the subsequent embodiments of the present disclosure is generally performed by the server 105 with stronger computing power and more computing resources. Accordingly, the corner point detection device is generally also set in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the above-mentioned operations that are handed over to the server 105 through the image processing applications installed thereon, and then output the same results as the server 105. In particular, when there are multiple terminal devices with different computing powers at the same time, but the image processing application determines that the terminal device where it is located has stronger computing power and more remaining computing resources, it can let the terminal device perform the above-mentioned operations, thereby appropriately reducing the computing pressure of the server 105. Accordingly, the corner point detection device can also be set in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104.

[0030] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0031] Please refer to Figure 2 , Figure 2 A flowchart of a corner point detection method provided by an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0032] Step 201: extracting the outer contour of the target object based on the image of the target object.

[0033] This step is intended to be performed by the execution body of the corner detection method (e.g. Figure 1 The server 105 shown in the figure obtains an image of the target object, and extracts the outer contour information of the target object based on the image. The target object refers to an object whose image can be obtained by photographing or scanning, such as a person, a document, a book, an ID card and other items. The image of the target object can be a photo taken by a camera or other photographing device, or an electronic image scanned by a scanner or other device. The execution subject can control the camera to complete the photo taking, or control the scanner to complete the scanning, so as to obtain the image of the target object, or it can obtain a pre-stored photo or electronic image from the cloud or other servers, and the present embodiment is not limited to this. The extraction of the outer contour of the target object can be achieved by the execution subject through image segmentation, target recognition and other computer vision technologies.

[0034] Step 202: Calculate the convex hull based on the points in the outer contour to obtain a convex hull point sequence.

[0035] This step is intended to enable the above-mentioned execution subject to calculate the convex hull based on the points in the outer contour of the target object, and to construct a convex hull point sequence based on the convex hull. The convex hull is the smallest convex set that contains a finite set of points in a given plane or space. Convex hulls are often used for target detection and shape analysis. In image processing, convex hulls can be used to extract the contours of objects in an image, thereby performing subsequent operations such as object recognition and tracking. The more mature algorithms in related technologies (such as Jarvis stepping method, etc.) can be used to calculate the convex hull and obtain a convex hull point sequence.

[0036] Step 203: Determine a target convex quadrilateral based on the convex hull point sequence.

[0037] This step aims to enable the above-mentioned execution subject to form a quadrilateral based on the points in the convex hull sequence, so as to obtain multiple combinations of four points and determine the target convex quadrilateral from the multiple combinations.

[0038] Step 204: Obtain the coordinates of the corner points of the target object based on the target convex quadrilateral.

[0039] After the target convex quadrilateral is determined, the execution subject can determine the corner coordinates of the target object according to the vertex coordinates of the target quadrilateral.

[0040] The corner point detection method provided by the embodiment of the present disclosure obtains a convex hull sequence based on the outer contour of the target object during the process of processing the image of the target object, and finds four points that can form a quadrilateral based on the points in the convex hull sequence. Since the points on the convex hull are arranged in a certain order (for example, counterclockwise or clockwise), the search process can be effectively simplified and the amount of calculation can be reduced. Moreover, the process of constructing a quadrilateral based on the convex hull can more effectively and quickly determine the target convex quadrilateral, and use it for subsequent calculations to determine the coordinates of the corner points of the target object, which can further improve the calculation efficiency.

[0041] In some optional implementations of this embodiment, the above step 202, the process of calculating the convex hull based on the points in the outer contour to obtain the convex hull point sequence, mainly includes: first, simplifying the points in the outer contour to obtain a simplified point set. In this embodiment, the points in the outer contour are simplified to reduce the complexity of the outer contour, and by removing unnecessary vertices, the contour is smoother and easier to handle. In a specific implementation, the execution subject can implement the simplified processing of the outer contour points through relatively mature technologies in related fields. Then, the convex hull is calculated based on the points in the simplified point set to obtain a convex hull point sequence. Through this process, the number of points that need to be calculated can be further simplified, the calculation overhead can be reduced, and the processing speed and efficiency can be improved. In practical applications, based on the above-mentioned optimized corner point detection method, since the processing speed is further improved and the amount of calculation is simplified, it can not only be limited to applications in devices with large computing power such as servers, but can also be directly deployed on user terminal devices (such as mobile phones, tablet computers, etc.), expanding the application scenarios of the solution and increasing universality.

[0042] In some optional implementations of this embodiment, please refer to Figure 3A , Figure 3A A flowchart of a process for determining a target convex quadrilateral based on a convex hull point sequence provided in an embodiment of the present disclosure, that is, for Figure 2 Step 203 in the process 200 shown provides a specific implementation method, and the other steps in the process 200 are not adjusted. The specific implementation method provided in this embodiment is replaced by step 203 to obtain a new complete embodiment. The process 300 includes the following steps:

[0043] Step 301: Traverse any combination of four points in the convex hull point sequence to construct a candidate quadrilateral.

[0044] Step 302: Calculate the area of ​​each candidate quadrilateral and determine the convex quadrilateral with the smallest area as the target convex quadrilateral.

[0045] In the above process, the execution subject selects any four points from the convex hull sequence to form a quadrilateral, so as to obtain multiple combinations of four points and their corresponding candidate quadrilaterals. Then, the area of ​​each candidate quadrilateral is calculated respectively, and the quadrilateral with the smallest area is selected from them. Moreover, the quadrilateral selected by the execution subject should belong to a convex quadrilateral, and the convex quadrilateral with the smallest area is determined as the target convex quadrilateral.

[0046] One of the main objectives of the corner point detection method of the disclosed embodiment is to reduce the amount of calculation as much as possible and improve the calculation efficiency without affecting the accuracy of the final result. Step 301 of the above embodiment, traversing any combination of four points in the convex hull point sequence to construct a candidate quadrilateral, can also be further optimized. Figure 3B As shown, step 301 mainly includes:

[0047] Step 3011: Select any combination of four points in the convex hull point sequence to construct multiple first candidate quadrilaterals.

[0048] Step 3012: Determine whether the angle between every two adjacent sides of each first candidate quadrilateral is not less than a minimum preset angle (eg, 30 degrees) and not greater than a maximum preset angle (eg, 120 degrees).

[0049] Step 3013: If the included angle between every two adjacent sides of each first candidate quadrilateral is not less than the minimum preset angle and not greater than the maximum preset angle, the first candidate quadrilateral that meets this condition is used as a candidate quadrilateral.

[0050] Step 3014: If the included angle between every two adjacent sides of each first candidate quadrilateral is smaller than the minimum preset angle, or larger than the maximum preset angle, skip the four points in the combination corresponding to the current first candidate quadrilateral and select another combination of any four points.

[0051] Through the above process, in the process of constructing candidate quadrilaterals, some combinations of points corresponding to shapes that may obviously not conform to the rules of conventional quadrilaterals are screened out, thereby further reducing the amount of data required for calculation in subsequent steps, so as to achieve the purpose of improving processing speed and improving computing efficiency.

[0052] In some optional implementations of this embodiment, the above step 302, the process of calculating the area of ​​each candidate quadrilateral and determining the convex quadrilateral with the smallest area as the target convex quadrilateral mainly includes: selecting candidate convex quadrilaterals belonging to convex figures from each candidate quadrilateral. Among them, a convex figure refers to a figure in which the line segments between any two points in a figure are completely located inside or on the boundary of the figure. The execution subject can use machine vision and other technologies to determine whether each candidate quadrilateral is a convex figure. Alternatively, for each pair of adjacent sides of each candidate quadrilateral, their intersection points can be calculated to form quadrilateral vertices, and whether the quadrilateral is a convex figure can be determined based on the positional relationship of the vertices. If it is a convex figure, it is retained, and if it is not a convex figure, the quadrilateral is skipped, thereby screening out candidate convex quadrilaterals. Then, the area of ​​each candidate convex quadrilateral is calculated, and the convex quadrilateral with the smallest area is determined to be the target convex quadrilateral.

[0053] The corner detection method provided by the embodiment of the present disclosure can be applied to a wide range of scenarios, such as motion detection, image matching, video tracking, target recognition, intelligent navigation, and visual positioning. In these application scenarios, it is not only required to improve the computing efficiency and reduce the delay, but also to have a relatively high requirement for recognition accuracy. In order to adapt to the recognition accuracy requirements of different application scenarios, in some optional implementation methods of this embodiment, please refer to Figure 4 , Figure 4 A flowchart of extracting the outer contour of a target object based on an image of the target object provided in an embodiment of the present disclosure, that is, for Figure 2 Step 201 in the process 200 shown provides a specific implementation method, and the other steps in the process 200 are not adjusted. The specific implementation method provided in this embodiment is replaced by step 201 to obtain a new complete embodiment. The process 400 includes the following steps:

[0054] Step 401: Acquire an original image of a target object.

[0055] In this embodiment, the original image may be a photo taken by a camera or other photographing device, or an electronic image scanned by a scanner or other device. The execution subject may control the camera to complete the photo taking, or control the scanner to complete the scanning, thereby obtaining the image of the target object, or may obtain a pre-stored photo or electronic image from the cloud or other servers, and this embodiment is not limited thereto.

[0056] Step 402: semantically segment the original image using a pre-trained semantic segmentation model to obtain a semantically segmented image.

[0057] The semantic segmentation model aims to classify each pixel in the image into a corresponding target category, thereby achieving a detailed understanding of the image content. The semantic segmentation model can distinguish different objects and backgrounds, and assign a label to each pixel to indicate the category to which it belongs. In this embodiment, the semantic segmentation model called by the execution subject can be implemented based on a semantic segmentation model that is already relatively mature in the relevant field (such as a fully convolutional network FCN, etc.). The semantic segmentation image obtained by the semantic segmentation model can be output in the form of a heatmap.

[0058] In this embodiment, based on this type of semantic segmentation model, a solution that is more in line with the requirements of this embodiment is proposed. During the training process of the semantic segmentation model, the category of the segmentation head is adjusted to the foreground to characterize the target area and the background to characterize the non-target area, so as to be more suitable for realizing the requirements of the solution of this embodiment. The target area refers to the area where the target object is located, and the non-target area refers to the area other than the target object. For example, when the target object is a document, the target area is the document, and the non-target area is the background part outside the document.

[0059] Furthermore, the first loss weight of the cross entropy loss corresponding to the target area is adjusted to be greater than the second loss weight of the cross entropy loss corresponding to the non-target area, so as to further highlight the segmentation of the target area. Furthermore, the first loss weight can be adjusted to twice the second loss weight.

[0060] Step 403: extracting the outer contour of the target object based on the semantic segmentation image.

[0061] After obtaining the semantic segmentation image, the execution subject can extract the outer contour of the target object based on the semantic segmentation image. For example, the outer contour can be obtained by performing contour extraction on the heat map.

[0062] In some optional implementations of this embodiment, step 403, the process of extracting the outer contour of the target object based on the semantic segmentation image, mainly includes:

[0063] Step 1: Perform erosion and dilation operations on the semantic segmentation image to obtain the processed image. The main function of the erosion and dilation operation is to eliminate or fill in some small defects and smooth the segmentation edges.

[0064] Step 2: Extract the largest connected domain in the processed image. In the processed image, the smaller connected domain is not the area where the target object is located by default, and it is noise. Therefore, the largest connected domain is extracted, that is, it is considered that the largest connected domain corresponds to the area where the target object is located.

[0065] Step 3: Extract the outer contour of the largest connected domain as the outer contour of the target object.

[0066] Through the above process, the process of semantic segmentation of the original image is adjusted and improved, which can improve the recognition accuracy of different areas in the original image and provide a more accurate basis for the subsequent corner detection process and application scenarios.

[0067] Please refer to Figure 5 , Figure 5 A flowchart of another corner point detection method provided by an embodiment of the present disclosure, wherein process 500 includes the following steps:

[0068] Step 501: extracting the outer contour of the target object based on the image of the target object.

[0069] Step 502: Calculate the convex hull based on the points in the outer contour to obtain a convex hull point sequence.

[0070] Step 503: Determine a target convex quadrilateral based on the convex hull point sequence.

[0071] Step 504: Obtain the coordinates of the corner points of the target object based on the target convex quadrilateral.

[0072] The above steps 501-504 are similar to Figure 2 Steps 201-204 shown are consistent, and for the same contents, please refer to the corresponding parts of the previous embodiment, which will not be repeated here.

[0073] Step 505: Perform perspective transformation based on the corner point coordinates of the target object to obtain a corrected image of the target object.

[0074] Perspective transformation refers to the transformation that utilizes the condition that the perspective center, the image point, and the target point are collinear, and according to the perspective rotation law, the image receiving surface (perspective surface) is rotated around the trace (perspective axis) by a certain angle, destroying the original projection light beam, but still keeping the projection geometry on the image receiving surface unchanged. Points in a two-dimensional coordinate system can be mapped to points in a three-dimensional coordinate system, and then projected to points in another two-dimensional coordinate system. This transformation is achieved through a transformation matrix, which acts on each pixel coordinate of the image, thereby performing coordinate mapping conversion. Image distortion caused by viewing angle can be corrected through perspective transformation, such as correcting a tilted book or building photo to a front view. Therefore, in this embodiment, based on the corner point coordinates of the target object obtained by the corner point detection method of any of the above embodiments, the image of the target object is corrected through perspective transformation to obtain a corrected image. In some optional implementations of this embodiment, step 506, the process of performing perspective transformation based on the corner point coordinates of the target object to obtain a corrected image of the target object, mainly includes: determining the aspect ratio of the target object based on the corner point coordinates; determining the length and width information of the target object based on the aspect ratio; performing perspective transformation based on the length and width information to obtain a corrected image.

[0075] The image correction process achieved by the corner detection method described in any of the above embodiments has a certain degree of improvement in the results compared to the related art. Taking the target object as a document as an example, the corner detection method described in the embodiment of the present disclosure is compared with some image correction schemes in the related art, as shown in Table 1 below, which shows the comparison results of the corner detection method described in the embodiment of the present disclosure and some image correction schemes in the related art (Comparison Scheme 1 and Comparison Scheme 2) for a set of 592 document images.

[0076] Table 1

[0077] product miou miou>0.9(pieces) miou>0.8(pieces) Comparison plan 1 0.945 534 554 Comparison plan 2 0.955 551 566 This public scheme 0.963 554 576

[0078] From Table 1, it can be seen that the indicator miou (miou is an evaluation indicator frequently used in the fields of machine learning and computer vision, especially in image segmentation tasks. It stands for "Mean Intersection over Union". This indicator is designed to quantify the similarity or accuracy between the segmentation results predicted by the evaluation model and the actual segmentation annotation) exceeds that of comparison scheme 1 and comparison scheme 2, and the number of miou greater than 0.9 and 0.8 also exceeds that of comparison scheme 1 and comparison scheme 2.

[0079] In order to more intuitively demonstrate the comparison effect, please refer to Figures 6A to 6C. In Figures 6A to 6C, from left to right are the original image, the effect image obtained by the related technology and the solution of the embodiment of the present disclosure (i.e. Figure 6A-1 , Figure 6B-1 , Figure 6C-1 The original image is shown; Figure 6A-2 , Figure 6B-2 , Figure 6C-2 The following is the effect diagram after the relevant technology processing; Figure 6A-3 , Figure 6B-3 , Figure 6C-3 6A to 6C, the mosaic marks presented for the text part are not the content existing in the image itself, but the desensitization processing performed in the embodiment of the present disclosure to avoid sensitive information. This part of the content is not the effect obtained by the processing of the solution of the embodiment of the present disclosure, nor is it the reference content used to show the difference in image processing.

[0080] To deepen understanding, the present disclosure also provides a specific implementation solution in combination with a specific application scenario, see Fig. 7A and Figure 7B shown.

[0081] In this embodiment, the target object is a document 71. The process 700 of correcting the image of the document 71 mainly includes:

[0082] Step 701: Obtain a heatmap 73 of the outer contour based on the original image 72 of the document 71 using a pre-trained semantic segmentation model.

[0083] Step 702: extract the outer contour of the document 71 based on the heat map 73 (erosion and dilation operation - extract the maximum connected domain - extract the outer contour).

[0084] Step 703: Perform simplification processing based on the points in the outer contour to obtain a simplified point set.

[0085] Step 704: Calculate the convex hull according to the points in the simplified point set to obtain a convex hull point sequence.

[0086] Step 705: Traverse any four point combinations in the convex hull point sequence to construct a candidate quadrilateral. Perform angle pruning on the candidate quadrilateral to check whether the angle between each two adjacent sides meets the specified minimum angle and maximum angle range conditions. If the conditions are not met, skip the current candidate point combination directly.

[0087] Step 706: If the conditions are met, determine whether each candidate quadrilateral is a convex figure. If it is a convex figure, calculate the area of ​​each candidate convex quadrilateral to determine the convex quadrilateral with the minimum area.

[0088] Step 707: Obtain the corner point coordinates 74 of the document 71 based on the convex quadrilateral with the minimum area.

[0089] Step 708 : Based on the corner point coordinates 74 , perform perspective transformation on the original image 71 of the document 71 to obtain a corrected image 75 of the document 71 .

[0090] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a corner point detection device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0091] like Figure 8 As shown, the corner point detection device 800 of this embodiment may include: an outer contour extraction module 801, a convex hull point sequence calculation module 802, a target quadrilateral determination module 803 and a corner point coordinate generation module 804. Among them, the outer contour extraction module 801 is configured to extract the outer contour of the target object based on the image of the target object. The convex hull point sequence calculation module 802 is configured to calculate the convex hull based on the points in the outer contour to obtain a convex hull point sequence. The target quadrilateral determination module 803 is configured to determine the target convex quadrilateral based on the convex hull point sequence. The corner point coordinate generation module 804 is configured to obtain the corner point coordinates of the target object based on the target convex quadrilateral.

[0092] In this embodiment, the specific processing of the outer contour extraction module 801, the convex hull point sequence calculation module 802, the target quadrilateral determination module 803 and the corner point coordinate generation module 804 and the technical effects thereof can be referred to in detail. Figure 2 The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0093] In some optional implementations of this embodiment, the convex hull point sequence calculation module 802 includes: a simplified point set generation submodule, configured to simplify the points in the outer contour to obtain a simplified point set; a convex hull point sequence generation submodule, configured to calculate the convex hull based on the points in the simplified point set to obtain a convex hull point sequence.

[0094] In some optional implementations of the present embodiment, the target quadrilateral determination module 803 includes: a candidate quadrilateral construction submodule, configured to traverse any combination of four points in the convex hull point sequence to construct a candidate quadrilateral; a convex quadrilateral determination submodule, configured to calculate the area of ​​each candidate quadrilateral and determine the convex quadrilateral with the smallest area as the target convex quadrilateral.

[0095] In some optional implementations of the present embodiment, the candidate quadrilateral construction submodule is further configured to: for a combination formed by any four points in the convex hull point sequence, in response to the angle between every two adjacent sides of the quadrilateral constructed by the four points of the combination being not less than the minimum preset angle and not greater than the maximum preset angle, take the quadrilateral constructed by the four points of the combination as a candidate quadrilateral; in response to the angle between every two adjacent sides of the quadrilateral constructed by the four points of the combination being less than the minimum preset angle, or greater than the maximum preset angle, skip the quadrilateral constructed by the four points of the combination.

[0096] In some optional implementations of this embodiment, the convex quadrilateral determination submodule includes: a candidate convex quadrilateral screening unit, configured to screen out candidate convex quadrilaterals belonging to convex figures from each candidate quadrilateral; a convex quadrilateral determination unit, configured to calculate the area of ​​each candidate convex quadrilateral and determine the convex quadrilateral with the smallest area.

[0097] In some optional implementations of the present embodiment, the outer contour extraction module 801 includes: an original image acquisition submodule, configured to acquire the original image of the target object; a semantic segmentation submodule, configured to perform semantic segmentation on the original image through a pre-trained semantic segmentation model to obtain a semantic segmentation image; and an outer contour extraction submodule, configured to extract the outer contour of the target object based on the semantic segmentation image. During the training process of the semantic segmentation model, the category of the segmentation head is adjusted to the foreground to characterize the target area and the background to characterize the non-target area, and the first loss weight of the cross entropy loss of the target area is adjusted to a second loss weight greater than the cross entropy loss of the non-target area. Exemplarily, the first loss weight is twice the second loss weight.

[0098] In some optional implementations of this embodiment, the semantic segmentation submodule includes: an image processing unit, configured to perform erosion and dilation operations on the semantic segmentation image to obtain a processed image; a maximum connected domain extraction unit, configured to extract the maximum connected domain in the processed image; and an outer contour extraction unit, configured to extract the outer contour of the maximum connected domain as the outer contour of the target object.

[0099] In some optional implementations of this embodiment, the corner point detection device 800 described in any of the above embodiments further includes: a correction image generation module, which is configured to perform perspective transformation based on the corner point coordinates of the target object to obtain a correction image of the target object.

[0100] In some optional implementations of this embodiment, the corrected image generation module includes: an aspect ratio determination submodule, configured to determine the aspect ratio of the target object based on the corner point coordinates; a length and width information determination submodule, configured to determine the length and width information of the target object based on the aspect ratio; and a corrected image generation submodule, configured to perform a perspective transformation based on the length and width information to obtain a corrected image.

[0101] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The further functional description of each module and unit is the same as that of the above-mentioned corresponding embodiment, and will not be repeated here. The corner point detection device provided by this embodiment obtains a convex hull sequence based on the outer contour of the target object in the process of processing the image of the target object, and finds four points that can form a quadrilateral based on the points in the convex hull sequence. Since the points on the convex hull are arranged in a certain order (for example, counterclockwise or clockwise), the search process can be effectively simplified and the amount of calculation can be reduced. Moreover, the process of constructing a quadrilateral based on the convex hull can also more effectively and quickly determine the target quadrilateral compared to the process of traversing all points to find the quadrilateral, and is used for subsequent calculations to determine the coordinates of the corner points of the target object, which can further improve the calculation efficiency.

[0102] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the corner point detection method described in any of the above embodiments can be implemented when the at least one processor executes.

[0103] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the corner point detection method described in any of the above embodiments when executed.

[0104] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which, when executed by a processor, can implement the corner point detection method described in any of the above embodiments.

[0105] Fig. 9A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0106] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0107] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0108] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the corner detection method. For example, in some embodiments, the corner detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the corner detection method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the corner detection method in any other appropriate manner (e.g., by means of firmware).

[0109] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0111] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0114] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0115] According to the technical solution of the embodiment of the present disclosure, in the process of processing the image of the target object, a convex hull sequence is obtained based on the outer contour of the target object, and the points in the convex hull sequence are traversed to find four points that can form a quadrilateral. Since the points on the convex hull are arranged in a certain order (for example, counterclockwise or clockwise), the search process can be effectively simplified and the amount of calculation can be reduced. Moreover, the process of constructing a quadrilateral based on the convex hull can more effectively and quickly determine the target quadrilateral, and use it for subsequent calculations to determine the coordinates of the corner points of the target object, which can further improve the calculation efficiency.

[0116] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0117] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A corner point detection method, comprising: Extracting an outer contour of the target object based on the image of the target object; Calculate the convex hull based on the points in the outer contour to obtain a convex hull point sequence; Determine a target convex quadrilateral based on the convex hull point sequence; The coordinates of the corner points of the target object are obtained based on the target convex quadrilateral.

2. The method according to claim 1, wherein: The step of calculating the convex hull based on the points in the outer contour to obtain a convex hull point sequence comprises: Simplifying the points in the outer contour to obtain a simplified point set; The convex hull is calculated based on the points in the simplified point set to obtain the convex hull point sequence.

3. The method according to claim 1, wherein: The determining of a target convex quadrilateral based on the convex hull point sequence comprises: Traversing any combination of four points in the convex hull point sequence to construct a candidate quadrilateral; The areas of the candidate quadrilaterals are calculated, and the convex quadrilateral with the smallest area is determined as the target convex quadrilateral.

4. The method according to claim 3, wherein: The traversing any combination of four points in the convex hull point sequence to construct a candidate quadrilateral includes: For a combination formed by any four points in the convex hull point sequence, in response to the fact that the angle between every two adjacent sides of a quadrilateral constructed by the four points of the combination is not less than a minimum preset angle and not greater than a maximum preset angle, the quadrilateral constructed by the four points of the combination is used as the candidate quadrilateral.

5. The method according to claim 4, wherein: The traversing any combination of four points in the convex hull point sequence to construct a candidate quadrilateral also includes: In response to the angle between every two adjacent sides of the quadrilateral constructed by the combined four points being smaller than the minimum preset angle or larger than the maximum preset angle, the quadrilateral constructed by the combined four points is skipped.

6. The method according to claim 3, wherein: The calculating the areas of the candidate quadrilaterals and determining the convex quadrilateral with the smallest area as the target convex quadrilateral includes: Selecting a candidate convex quadrilateral belonging to a convex figure from the candidate quadrilaterals; The areas of the candidate convex quadrilaterals are calculated, and the convex quadrilateral with the smallest area is determined as the target convex quadrilateral.

7. The method according to claim 1, wherein: The step of extracting the outer contour of the target object based on the image of the target object comprises: Acquire an original image of the target object; Performing semantic segmentation on the original image using a pre-trained semantic segmentation model to obtain a semantic segmentation image; The outer contour of the target object is extracted based on the semantic segmentation image.

8. The method according to claim 7, wherein: During the training process of the semantic segmentation model, the category of the segmentation head is adjusted to a foreground for characterizing a target area and a background for characterizing a non-target area, and a first loss weight of a cross entropy loss of the target area is adjusted to a second loss weight greater than the cross entropy loss of the non-target area.

9. The method according to claim 8, wherein: The first loss weight is twice the second loss weight.

10. The method according to claim 7, wherein: The step of extracting the outer contour of the target object based on the semantic segmentation image comprises: Performing an erosion and dilation operation on the semantic segmentation image to obtain a processed image; Extracting the largest connected domain in the processed image; The outer contour of the maximum connected domain is extracted as the outer contour of the target object.

11. The method according to any one of claims 1 to 10, further comprising: A perspective transformation is performed based on the corner point coordinates of the target object to obtain a corrected image of the target object.

12. The method according to claim 11, wherein: The performing perspective transformation based on the corner point coordinates of the target object to obtain a corrected image of the target object includes: Determine the aspect ratio of the target object based on the corner point coordinates; Determining length and width information of the target object based on the aspect ratio; A perspective transformation is performed based on the length and width information to obtain the corrected image.

13. A corner point detection device, comprising: An outer contour extraction module is configured to extract an outer contour of the target object based on an image of the target object; A convex hull point sequence calculation module is configured to calculate the convex hull based on the points in the outer contour to obtain a convex hull point sequence; a target quadrilateral determination module, configured to determine a target convex quadrilateral based on the convex hull point sequence; The corner point coordinate generation module is configured to obtain the corner point coordinates of the target object based on the target convex quadrilateral.

14. The device according to claim 13, wherein: The convex hull point sequence calculation module includes: A simplified point set generation submodule is configured to simplify the points in the outer contour to obtain a simplified point set; The convex hull point sequence generation submodule is configured to calculate the convex hull based on the points in the simplified point set to obtain the convex hull point sequence.

15. The device according to claim 13, wherein: The target quadrilateral determination module comprises: A candidate quadrilateral construction submodule is configured to traverse any combination of four points in the convex hull point sequence to construct a candidate quadrilateral; The convex quadrilateral determination submodule is configured to calculate the area of ​​each of the candidate quadrilaterals and determine the convex quadrilateral with the smallest area as the target convex quadrilateral.

16. The device according to claim 15, wherein: The candidate quadrilateral construction submodule is further configured to: For a combination formed by any four points in the convex hull point sequence, in response to the fact that the angle between every two adjacent sides of a quadrilateral constructed by the four points of the combination is not less than a minimum preset angle and not greater than a maximum preset angle, the quadrilateral constructed by the four points of the combination is used as the candidate quadrilateral.

17. The device according to claim 16, wherein: The candidate quadrilateral construction submodule is further configured to: In response to the angle between every two adjacent sides of the quadrilateral constructed by the combined four points being smaller than the minimum preset angle or larger than the maximum preset angle, the quadrilateral constructed by the combined four points is skipped.

18. The device according to claim 15, wherein: The convex quadrilateral determination submodule comprises: A candidate convex quadrilateral screening unit is configured to screen out a candidate convex quadrilateral belonging to a convex figure from the candidate quadrilaterals; The convex quadrilateral determination unit is configured to calculate the area of ​​each of the candidate convex quadrilaterals and determine the convex quadrilateral with the smallest area as the target convex quadrilateral.

19. The device according to claim 13, wherein: The outer contour extraction module comprises: An original image acquisition submodule, configured to acquire an original image of the target object; A semantic segmentation submodule is configured to perform semantic segmentation on the original image using a pre-trained semantic segmentation model to obtain a semantic segmentation image; The outer contour extraction submodule is configured to extract the outer contour of the target object based on the semantic segmentation image.

20. The device according to claim 19, wherein During the training process of the semantic segmentation model, the category of the segmentation head is adjusted to a foreground for characterizing a target area and a background for characterizing a non-target area, and a first loss weight of a cross entropy loss of the target area is adjusted to a second loss weight greater than the cross entropy loss of the non-target area.

21. The device according to claim 20, wherein: The first loss weight is twice the second loss weight.

22. The device according to claim 19, wherein The semantic segmentation submodule includes: An image processing unit is configured to perform an erosion and dilation operation on the semantic segmentation image to obtain a processed image; A maximum connected region extraction unit is configured to extract the maximum connected region in the processed image; The outer contour extraction unit is configured to extract the outer contour of the maximum connected domain as the outer contour of the target object.

23. The apparatus according to any one of claims 13 to 22, further comprising: The corrected image generation module is configured to perform perspective transformation based on the corner point coordinates of the target object to obtain a corrected image of the target object.

24. The device according to claim 23, wherein: The corrected image generation module comprises: an aspect ratio determination submodule, configured to determine the aspect ratio of the target object based on the corner point coordinates; A length and width information determination submodule, configured to determine the length and width information of the target object based on the length and width ratio; The corrected image generation submodule is configured to perform perspective transformation based on the length and width information to obtain the corrected image.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the corner detection method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the corner detection method according to any one of claims 1 to 12.

27. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the corner detection method according to any one of claims 1 to 12 are implemented.

Images