Image correction method and device, equipment, storage medium and program product

Abstract

The invention discloses an image correction method and device, equipment, a storage medium and a program product, and relates to the technical field of image processing, and the method comprises the steps: determining corresponding second corner points according to four first corner points of a detected target object, and taking the determined four second angular points as target angular points for correcting the four first angular points, determining a mapping relation, and correcting the target object to obtain a corrected image, thereby preventing the corrected image from severely stretching or compressing in the height direction, effectively reducing the probability that the content in the image is distorted, and improving the image quality. And the accuracy of subsequent image processing is ensured.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image correction method, apparatus, device, storage medium, and program product. Background Technology

[0002] In some scenarios, electronic devices capture images via cameras and process these images to complete specific tasks. When a camera on an electronic device takes a picture, its normal is usually not perfectly perpendicular to the horizontal plane of the object being photographed, inevitably resulting in some distortion in the captured image. To ensure the accuracy of subsequent tasks, the captured image usually needs to be corrected. However, current image correction methods are prone to scaling distortion and have poor correction results. Summary of the Invention

[0003] In view of the above problems, this application provides an image correction method, apparatus, device, storage medium, and program product to improve the image correction effect. The specific solution is as follows:

[0004] The first aspect of this application provides an image correction method, comprising:

[0005] Corner detection is performed on the target image to obtain the four first corner points of the target object in the target image;

[0006] Determine the second corner points corresponding to each first corner point, wherein the aspect ratio of the rectangle with the four second corner points as vertices is the ratio of the target height to the target width; the target width is the smaller of the length of the first line segment between the upper left and upper right first corner points and the length of the second line segment between the lower left and lower right first corner points; the target height is greater than or equal to the minimum of the first and second vertical distances, and less than or equal to the average of the first and second vertical distances; the first vertical distance is the vertical distance from the upper left first corner point to the second line segment, and the second vertical distance is the vertical distance from the upper right first corner point to the second line segment;

[0007] The target object is corrected based on the mapping relationship from the four first corner points to the four second corner points to obtain the corrected image.

[0008] In one possible implementation, determining the second corner point corresponding to each first corner point includes:

[0009] The greater of the lengths of the first line segment and the second line segment is determined as the width of the rectangle;

[0010] The height of the rectangle is obtained by multiplying the ratio of the larger length to the smaller length by the target height.

[0011] The second corner point corresponding to each first corner point is determined based on the width and height of the rectangle.

[0012] In one possible implementation, determining the second corner point corresponding to each first corner point includes:

[0013] The width of the rectangle is determined based on the computing resource utilization rate; the width of the rectangle is negatively correlated with the computing resource utilization rate.

[0014] The height of the rectangle is determined based on its width and its aspect ratio.

[0015] The second corner point corresponding to each first corner point is determined based on the width and height of the rectangle.

[0016] In one possible implementation, determining the second corner point corresponding to each first corner point includes:

[0017] Obtain the pre-configured width of the rectangle;

[0018] The height of the rectangle is determined based on its width and its aspect ratio.

[0019] The second corner point corresponding to each first corner point is determined based on the width and height of the rectangle.

[0020] In one possible implementation, determining the second corner point corresponding to each first corner point includes:

[0021] Multiple image examples are determined based on the ratio of the target height to the target width, and the aspect ratio of each image example is the ratio of the target height to the target width; the widths of different image examples are different.

[0022] Showing examples of various images;

[0023] If a selection is obtained for any image example, the four vertices of the image example are determined as the four second corner points corresponding to the four first corner points.

[0024] In one possible implementation, the target image is an image of a book; the corner detection of the target image includes:

[0025] Corner detection is performed on the complete page in the target image to obtain the four first corner points of the complete page.

[0026] A second aspect of this application provides an image correction apparatus, comprising:

[0027] The corner detection module is used to perform corner detection on the target image to obtain the four first corners of the target object in the target image;

[0028] The corner point determination module is used to determine the second corner points corresponding to each first corner point. The aspect ratio of the rectangle with the four second corner points as vertices is the ratio of the target height to the target width. The target width is the smaller of the lengths of the first line segment between the upper left and upper right first corner points and the second line segment between the lower left and lower right first corner points. The target height is greater than or equal to the minimum of the first and second vertical distances, and less than or equal to the average of the first and second vertical distances. The first vertical distance is the vertical distance from the upper left first corner point to the second line segment, and the second vertical distance is the vertical distance from the upper right first corner point to the second line segment.

[0029] The correction module is used to correct the target object based on the mapping relationship from the four first corner points to the four second corner points, so as to obtain the corrected image.

[0030] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the image correction method described in the first aspect or any implementation thereof.

[0031] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0032] The memory is used to store computer programs;

[0033] The processor is used to execute the computer program so that the electronic device can implement the image correction method of the first aspect or any implementation thereof.

[0034] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to perform the image correction method described in the first aspect or any implementation thereof.

[0035] Using the above technical solution, the image correction method, apparatus, device, storage medium, and program product provided in this application perform corner detection on a target image to obtain four first corner points of the target object in the target image. Based on these four first corner points, second corner points corresponding to each first corner point are determined. The aspect ratio of the rectangle with the four second corner points as vertices is the ratio of the target height to the target width. The target width is the smaller of the length of the first line segment between the upper left and upper right first corner points and the length of the second line segment between the lower left and lower right first corner points. The target height is greater than or equal to the minimum of the first and second vertical distances, and less than or equal to the average of the first and second vertical distances. The first vertical distance is the vertical distance from the upper left first corner point to the second line segment, and the second vertical distance is the vertical distance from the upper right first corner point to the second line segment. Based on the mapping relationship from the four first corner points to the four second corner points, the target object is corrected to obtain the corrected image. The image correction method provided in this application determines the corresponding second corner points based on the four first corner points of the detected target object, and uses the determined four second corner points as target corner points to determine the mapping relationship for correcting the four first corner points to correct the target object, thereby obtaining a corrected image. This avoids severe stretching or compression in the height direction of the corrected image, effectively reduces the probability of distortion in the image content, and ensures the accuracy of subsequent image processing. Attached Figure Description

[0036] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0037] Figure 1a An example diagram showing the situation where the normal of the camera is not perpendicular to the plane of the object being photographed when the camera is capturing images provided in this application;

[0038] Figure 1b Provided for this application Figure 1a The image shown is an example of the four corner points of the object captured by the camera.

[0039] Figure 2 A flowchart illustrating one implementation of the image correction method provided in this application;

[0040] Figure 3 An example image showing the four first corner points obtained by corner detection of the target image provided in this application;

[0041] Figure 4 A flowchart illustrating one implementation of determining the second corner point corresponding to each first corner point, as provided in this application;

[0042] Figure 5 Another flowchart for determining the second corner point corresponding to each first corner point provided in this application;

[0043] Figure 6 Another flowchart for determining the second corner point corresponding to each first corner point provided in this application;

[0044] Figure 7 An example image taken by the picture book reading robot provided in this application;

[0045] Figure 8 The image correction method provided in this application is based on existing image correction methods. Figure 7 The image shown is the result after correction;

[0046] Figure 9 The image correction method based on this application provided in this application... Figure 7 The image shown is the result after correction;

[0047] Figure 10 Another example image of the picture book reading robot provided in this application;

[0048] Figure 11 The image correction method provided in this application is based on existing image correction methods. Figure 10 The image shown is the result after correction;

[0049] Figure 12 The image correction method based on this application provided in this application... Figure 10 The image shown is the result after correction;

[0050] Figure 13 A schematic diagram of the image correction device provided in this application;

[0051] Figure 14 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0052] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0053] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0054] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0055] The inventors of this application have discovered that current image correction methods directly use four default corner points as target corner points for the detected four corner points to determine the mapping relationship (e.g., transformation matrix), and perform image correction based on this mapping relationship. The corrected image produced by this method will exhibit severe stretching or compression in the height direction, i.e., image scaling distortion, thus affecting subsequent processing and judgment of image content.

[0056] To improve the image correction effect, the proposed solution is presented in this application.

[0057] To better understand this application, some of the research processes involved will be explained first.

[0058] To address the aforementioned technical issues, this application has made further research findings, please refer to [link / reference]. Figures 1a-1b , Figure 1a This is an example diagram showing a situation where the normal of the camera is not perpendicular to the plane containing the object being photographed (e.g., a page of a book, which is rectangular) when the camera is capturing images according to an embodiment of this application. Figure 1b Provided for the embodiments of this application Figure 1a The image shown is an example of the four corner points of the subject captured by the camera (referred to as the first corner point for ease of description and distinction). Figure 1a In the example shown, the camera is tilted in only one direction (i.e., tilted in the front-to-back direction, away from the top of the subject and close to the bottom of the subject), so the area formed by the four corners of the subject in the image is trapezoidal (or similar to a trapezoid), as shown. Figure 1b The four corner points shown are (p1, p2, p3, p4).

[0059] Assuming the camera's normal is perpendicular to the plane of the object being photographed, and the camera is not tilted left or right, the tilt angle of the camera's normal (denoted as α) is 0 degrees. In the image captured at this time, the length of the upper edge of the object (i.e., the line segment from the upper left corner point p1 to the upper right corner point p2) (denoted as w1) is equal to the length of the lower edge (i.e., the line segment from the lower left corner point p4 to the lower right corner point p3) (denoted as w2), and the height from the upper left corner point to the lower edge (i.e., the vertical distance from the upper left corner point to the lower edge, denoted as h1) is equal to the height from the upper right corner point to the lower edge (i.e., the vertical distance from the upper right corner point to the lower edge, denoted as h2). At this point, the image can be considered a front view of the subject. The height of the subject in the image (h = (h1 + h2) / 2 = h1 = h2) can be considered to represent the true height of the subject. The aspect ratio is: ratio = h / w1 = h / w2. Under this aspect ratio, the image will not have distortion problems such as stretching or compression.

[0060] When the normal of the camera is tilted in the front-to-back direction, such as Figures 1a-1b As shown, as the tilt angle α increases, w1 / w2 gradually decreases, and h = (h1 + h2) / 2 also gradually decreases. In other words, w1 / w2 is negatively correlated with the tilt angle α. Similarly, h is negatively correlated with the tilt angle α. From this, we can deduce that h is positively correlated with w1 / w2 (i.e., the larger w1 / w2 is, the larger h is), or that h is negatively correlated with w2 / w1 (which represents the tilt of the camera) (i.e., the larger w2 / w1 is, the smaller h is).

[0061] As an example, cosα = F(w1 / w2). F() represents the mapping relationship between cosα and w1 / w2 (it can be a linear or non-linear relationship). In practical applications, α is usually an acute angle, therefore, w1 / w2 is positively correlated with cosα.

[0062] As an example, cosα = G(h). G() represents the mapping relationship (which can be linear or non-linear) between cosα and the height h = (h1 + h2) / 2 of the object in the image. In practical applications, α is usually an acute angle, therefore, h is positively correlated with cosα.

[0063] like Figure 1a As shown, according to the Pythagorean theorem, the actual height of the photographed object is H = h / cosα = h / F(w1 / w2) = h × F(w2 / w1).

[0064] Clearly, there is a correlation between the actual height of the photographed object and the ratio of the top and bottom edges of the image. Based on this discovery, this application is filed.

[0065] like Figure 2As shown in the figure, it is a flowchart of an implementation of the image correction method provided by the embodiment of the present application, which may include:

[0066] Step S201: Perform corner detection on the target image to obtain four corner points of the target object in the target image (for the convenience of description and distinction, denoted as the first corner points).

[0067] The target image can be an image captured in real time by a camera or an image read from a storage unit.

[0068] Step S202: Determine the second corner points corresponding to each of the first corner points.

[0069] Among them, each first corner point corresponds to a second corner point. Therefore, the four first corner points correspond to four second corner points. The area formed by connecting the four second corner points in sequence is a rectangular area. The aspect ratio of the rectangle with the four second corner points as vertices (that is, the aspect ratio of the corrected image) is the ratio of the target height (denoted as h0) to the target width (denoted as w0) (i.e., h0 / w0). That is to say, the four determined second corner points are the four vertices of a rectangle. The height of this rectangle (denoted as h) can be the target height h0 or not, and similarly, the width of this rectangle (denoted as w) can be the target width w0 or not, but the ratio of the height h and width w of this rectangle must be equal to the ratio of the target height h0 and target width w0.

[0070] As Figure 3 shown in the figure, it is an example diagram of the four first corner points obtained by performing corner detection on the target image provided by the embodiment of the present application. In this example, the four first corner points are the upper left first corner point p1, the upper right first corner point p2, the lower right first corner point p3, and the lower left first corner point p4 respectively.

[0071] The target width w0 is the smaller length between the length of the first line segment (i.e., w1) between the upper left first corner point and the upper right first corner point and the length of the second line segment (i.e., w2) between the lower left first corner point and the lower right first corner point. That is to say, the target width w0 is the minimum value of w1 and w2. Figure 3 In the shown example, w1 < w2. Therefore, the target width w0 is w1.

[0072] The target height h0 is greater than or equal to the minimum value of the first vertical distance (i.e., h1) and the second vertical distance (i.e., h2), and less than or equal to the average value of the first vertical distance (i.e., h1) and the second vertical distance (i.e., h2); the first vertical distance (i.e., h1) is the vertical distance from the first upper left corner point p1 to the second line segment, and the second vertical distance (i.e., h2) is the vertical distance from the first upper right corner point p2 to the second line segment. That is to say, the minimum value of the target height h0 is the minimum value of h1 and h2, and the maximum value of the target height h0 is the average value of h1 and h2.

[0073] Take Figure 3 as an example, the aspect ratio of the rectangle with the four second corner points as vertices can be ((h1 + h2) / 2) / w1. In this example, the target height h0 = (h1 + h2) / 2.

[0074] Or Figure 3 In, the minimum value of h1 and h2 is h1, and the minimum value of w1 and w2 is w1, then the target height h0 can be h1. Therefore, the aspect ratio of the rectangle with the four second corner points as vertices can be h1 / w1. Since in the actual shooting process, the lens cannot only have the tilt angle in the front and back directions, but there will also be the tilt angle in the left and right directions, so the target height h0 can select the minimum value of the first vertical distance h1 from the first upper left corner point of the target object to the lower side and the second vertical distance h2 from the first upper right corner point to the lower side, which can prevent the converted image from being overstretched in the height direction.

[0075] Or, the aspect ratio of the rectangle with the four second corner points as vertices can be h3 / w1. Among them, h1 < h3 < (h1 + h2) / 2. In this example, the target height h0 = h3.

[0076] In this application, the target height and target width can be determined first according to the coordinates of the four first corner points, and then the height of the rectangle can be determined according to the target height and target width, as well as the obtained width of the rectangle, and then the four second corner points can be determined. Among them, the width of the rectangle can be automatically obtained according to the running situation of the electronic device, or can be pre-configured by the user.

[0077] Or, the width and height of the rectangle can be directly determined according to the coordinates of the four first corner points, and then the four second corner points can be determined.

[0078] After determining the width and height of the rectangle, one of the second corner points can be used as the origin of the image coordinate system, and the coordinates of the other second corner points can be determined based on the width and height of the rectangle.

[0079] Step S203: Correct the target object based on the mapping relationship from the four first corner points to the four second corner points to obtain the corrected image.

[0080] Image correction can be achieved through perspective transformation, a type of projection transformation that converts an image from one viewpoint to another. When capturing images, due to the shooting angle (different shooting angles are different viewpoints), the image may appear tilted or distorted. Perspective transformation can correct the image to a frontal view, facilitating subsequent processing and analysis.

[0081] Perspective transformation can be achieved using a transformation matrix (denoted as M). The transformation matrix M can be a 3×3 matrix, as shown below:

[0082]

[0083] For any point in the target image, let its coordinates be (x, y). Assuming the coordinates of a point in the corrected image after perspective transformation are (x', y'), the calculation relationship between these two points is as follows:

[0084]

[0085] According to formula (2), we can deduce that:

[0086]

[0087] As shown in formula (1), the transformation matrix M has 8 unknowns, requiring at least four pairs of coordinate points for solution. In this application, the four pairs of coordinate points are the four detected first corner points and the corresponding four second corner points. Among these four second corner points, the line segment between the upper left and upper right second corner points is parallel to the line segment between the lower left and lower right second corner points; the line segment between the upper left and lower left second corner points is perpendicular to the line segment between the upper left and upper right second corner points; and the line segment between the upper right and lower right second corner points is perpendicular to the line segment between the upper left and upper right second corner points.

[0088] The coordinates of the first and second corner points at the top left form a pair of coordinate points; the coordinates of the first and second corner points at the top right form a pair of coordinate points; the coordinates of the first and second corner points at the bottom left form a pair of coordinate points; and the coordinates of the first and second corner points at the bottom right form a pair of coordinate points.

[0089] For each pair of coordinate points, substitute the coordinates (x, y) of the first corner point and the coordinates (x', y') of the second corner point into formula (3). Then, a total of 8 linear equations are obtained from the four pairs of coordinate points. Solve the 8 linear equations to obtain the values ​​of the 8 unknowns.

[0090] As an example, a pre-defined transformation matrix calculation interface can be called to calculate the transformation matrix. For instance, the `cv2.getPerspectiveTransform(src_pts, dst_pts)` interface in OpenCV can be called to calculate the transformation matrix M. Here, `src_pts` represents the coordinates of the four first corner points, and `dst_pts` represents the coordinates of the four second corner points.

[0091] After calculating the transformation matrix M, a preset correction interface can be called to correct the target image. For example, the OpenCV interface `warped = cv2.warpPerspective(image, M, (width, height))` can be used to correct the target object. Here, `image` is the region containing the target object cropped from the target image (i.e., the quadrilateral region enclosed by the four first corner points), `M` is the transformation matrix, `width` is the width of the corrected image (i.e., the straight-line distance between the top-left and top-right second corner points), and `height` is the height of the corrected image (i.e., the straight-line distance between the top-left and bottom-left second corner points).

[0092] The image correction method provided in this application determines the corresponding second corner points based on the four first corner points of the detected target object, and uses the determined four second corner points as target corner points to determine the mapping relationship for correcting the four first corner points to correct the target object, thereby obtaining a corrected image. This avoids severe stretching or compression in the height direction of the corrected image, effectively reduces the probability of distortion in the image content, and ensures the accuracy of subsequent image processing.

[0093] In an optional embodiment, a flowchart illustrating one method for determining the second corner point corresponding to each first corner point is shown below. Figure 4 As shown, it may include:

[0094] Step S401: Determine the width of the rectangle as the larger of the lengths of the first and second line segments.

[0095] by Figure 3 For example, since the length of the first line segment w1 is less than the length of the second line segment w2, the length of the second line segment w2 is determined to be the width of the rectangle (i.e. the width of the corrected image).

[0096] Step S402: Multiply the ratio of the larger length to the smaller length by the target height to obtain the height of the rectangle (i.e., the height of the corrected image).

[0097] by Figure 3 For example, the height of the rectangle is h0×(w2 / w1). Based on this, the aspect ratio of the rectangle is h0 / w1.

[0098] Step S403: Determine the second corner point corresponding to each first corner point based on the width and height of the rectangle.

[0099] The first corner point at the top left corresponds to the second corner point at the top left, with coordinates (0, 0); the first corner point at the top right corresponds to the second corner point at the top right, with coordinates (w2, 0); the first corner point at the bottom left corresponds to the second corner point at the bottom left, with coordinates (0, h0×(w2 / w1)); and the first corner point at the bottom right corresponds to the second corner point at the bottom right, with coordinates (w2, h0×(w2 / w1)).

[0100] In this embodiment, the width and height of the corrected image (i.e., rectangle) are calculated directly based on the relative positional relationship of the four first corner points (the length of the line segment between the first corner points, the vertical distance between the line segments between the corner points, etc.). The size of the corrected image is more in line with the user's actual needs, and there will be no excessive stretching or compression in height.

[0101] In an optional embodiment, another implementation flowchart for determining the second corner points corresponding to each first corner point is shown below. Figure 5 As shown, it may include:

[0102] Step S501: Determine the width of the rectangle (that is, the width of the corrected image) based on the computational resource occupancy rate; the width of the rectangle is negatively correlated with the computational resource occupancy rate.

[0103] The image correction method of this application is applied to an electronic device. The computing resources of the electronic device may include, but are not limited to, at least one of the following: a central processing unit (CPU), a graphics processing unit (GPU), and memory. Specifically, the width of the rectangle is negatively correlated with the CPU utilization rate, the GPU utilization rate, and the memory utilization rate of the electronic device. That is, the higher the CPU utilization rate, the smaller the width of the rectangle; the higher the GPU utilization rate, the smaller the width of the rectangle; and the higher the memory utilization rate, the smaller the width of the rectangle.

[0104] The width of the rectangle can be determined by the real-time computing resource utilization rate of the electronic device and the preset correspondence between the computing resource utilization rate and the rectangle width.

[0105] Step S502: Determine the height of the rectangle based on its width and aspect ratio.

[0106] Multiply the width of the rectangle by its aspect ratio to obtain the height of the rectangle. The aspect ratio can be obtained before step S502, or it can be obtained after or before step S501.

[0107] Step S503: Determine the second corner point corresponding to each first corner point based on the width and height of the rectangle.

[0108] For details on the specific implementation method, please refer to step S403, which will not be elaborated here.

[0109] This embodiment first calculates the aspect ratio of the corrected image, then determines the width of the corrected image based on the utilization of computing resources, and finally determines the height of the corrected image based on its width and aspect ratio. Based on this application, a higher resolution image can be obtained when computing resources are sufficient (lower utilization), and a lower resolution image can be obtained when computing resources are insufficient (higher utilization), thus improving image correction effect while ensuring correction efficiency.

[0110] In an optional embodiment, another implementation flowchart for determining the second corner points corresponding to each first corner point is shown below. Figure 6 As shown, it may include:

[0111] Step S601: Obtain the width of the pre-configured rectangle.

[0112] In this embodiment, the user can manually configure the width of the corrected image as needed.

[0113] As an example, multiple image examples with different aspect ratios (the ratio of the target height to the target width) and varying widths can be output, allowing the user to select the rectangle's width based on these image examples. The width of any given image example can be determined as the rectangle's width in response to the user's selection.

[0114] As an example, users can directly enter the width of the corrected image in the image width configuration box.

[0115] Step S602: Determine the height of the rectangle based on its width and aspect ratio.

[0116] Multiply the width of the rectangle by its aspect ratio to obtain the height of the rectangle. The aspect ratio can be obtained before step S502, after step S501, or between steps S501.

[0117] Optionally, steps S601 and S602 can be combined into one step, that is, the height of the image example selected by the user can be directly determined as the height of the rectangle.

[0118] Step S603: Determine the second corner point corresponding to each first corner point based on the width and height of the rectangle.

[0119] For details, please refer to step S403, which will not be repeated here.

[0120] In this embodiment, the width of the corrected image can be manually configured by the user, so that the size of the corrected image better meets the user's needs.

[0121] In an optional embodiment, another way to determine the second corner point corresponding to each first corner point can be:

[0122] Multiple image examples are determined based on the ratio of the target height to the target width. The aspect ratio of each image example is the ratio of the target height to the target width; the width of different image examples is different.

[0123] Display various image examples. These examples can be displayed sequentially, one at a time, allowing users to easily identify the size of different image examples. Image example switching can be automatic (e.g., switching every preset time interval) or triggered by a user-initiated switching command.

[0124] If a selection is obtained for any image example, the four vertices of that image example are determined as the four second corner points corresponding to the four first corner points.

[0125] In an optional embodiment, the target image is an image of a book. Accordingly, one implementation of the above-described corner detection of the target image can be:

[0126] Corner detection is performed on the complete page in the target image to obtain the four first corner points of the complete page.

[0127] like Figure 7 The image shown is an example of an image captured by the picture book reading robot provided in this application embodiment. In this example, the complete content of page 51 of the picture book and a portion of page 50 are captured. Based on this, this application... Figure 7 The image shown was subjected to corner detection, which yielded four corner points on page 51. That is, the quadrilateral area enclosed by the four detected first corner points only covers page 51, but not page 50.

[0128] Figure 7 The actual size of the image shown is 5569×7394. According to actual measurements, the actual page height of the picture book is 26.1cm, the width is 17.8cm, and the aspect ratio is approximately 1.466.

[0129] Existing image correction methods directly determine the four second corner points corresponding to the four first corner points based on the true aspect ratio of the picture book (1.466) (i.e., the aspect ratio of the rectangle determined by the four second corner points is 1.466) and perform perspective transformation, resulting in a corrected image as shown below. Figure 8 As shown, the corrected image has a height of 6226, a width of 3125, and an aspect ratio of 1.992. The corrected image exhibits a significant stretching distortion in the height direction.

[0130] Based on the image correction method of this application, the corrected image is as follows: Figure 9 As shown, the corrected image has a height of 4730, a width of 3144, and an aspect ratio of 1.504. Clearly, the aspect ratio of the corrected image obtained using the image correction method of this application is closer to that of the actual picture book, effectively alleviating the problem of scale distortion.

[0131] like Figure 10 The image shown is another example of an image captured by the picture book reading robot provided in this application embodiment. In this example, the complete content of page 33 of the picture book and a portion of page 29 were captured. Based on this, this application... Figure 10 The image shown was subjected to corner detection, which yielded four corner points on page 33. That is, the quadrilateral area enclosed by the four detected first corner points only covers page 33, but does not cover page 29.

[0132] Figure 10 The actual size of the image shown is 5560×7394. According to actual measurements, the actual page height of the picture book is 26.1cm, the width is 17.6cm, and the aspect ratio is approximately 1.48.

[0133] Existing image correction methods directly determine the four second corner points corresponding to the four first corner points based on the true aspect ratio of the picture book (1.48) (i.e., the aspect ratio of the rectangle determined by the four second corner points is 1.48) and perform perspective transformation, resulting in a corrected image as shown below. Figure 11 As shown, the corrected image has a height of 3470, a width of 4633, and an aspect ratio of 0.749. The corrected image exhibits a significant compression distortion in the height direction.

[0134] Based on the image correction method of this application, for Figure 10 The corrected image shown is as follows: Figure 12 As shown, the corrected image has a height of 5788, a width of 4665, and an aspect ratio of 1.241. Clearly, the aspect ratio of the corrected image obtained using the image correction method of this application is closer to that of the actual picture book, effectively alleviating the problem of scale distortion.

[0135] Optionally, a pre-trained corner detection model can be used to detect the target image and obtain the four first corners of the complete page.

[0136] The corner detection model is trained using book images as training samples and the coordinate information of the four corner points of a complete page in the book image as sample labels.

[0137] As an example, the corner detection model can be a YOLO model, such as a YOLOv8 model or a YOLOv11 model. Of course, the corner detection model can also be a model with other architectures, and this application does not make any specific restrictions.

[0138] Corresponding to the method embodiments, this application also provides an image correction device. A schematic diagram of the structure of the image correction device provided in this application embodiment is shown below. Figure 13 As shown, it may include:

[0139] Corner detection module 1301, corner determination module 1302 and correction module 1303;

[0140] The corner detection module 1301 is used to perform corner detection on the target image to obtain the four first corners of the target object in the target image;

[0141] The corner point determination module 1302 is used to determine the second corner points corresponding to each first corner point, wherein the aspect ratio of the rectangle with the four second corner points as vertices is the ratio of the target height to the target width; the target width is the smaller of the length of the first line segment between the upper left and upper right first corner points and the length of the second line segment between the lower left and lower right first corner points; the target height is greater than or equal to the minimum of the first vertical distance and the second vertical distance, and less than or equal to the average of the first vertical distance and the second vertical distance; the first vertical distance is the vertical distance from the upper left first corner point to the second line segment, and the second vertical distance is the vertical distance from the upper right first corner point to the second line segment;

[0142] The correction module 1303 is used to correct the target object based on the mapping relationship from the four first corner points to the four second corner points, so as to obtain the corrected image.

[0143] The image correction device provided in this application determines the corresponding second corner points based on the four first corner points of the detected target object. The four determined second corner points are used as target corner points to determine the mapping relationship for correcting the four first corner points, thereby correcting the target object and obtaining a corrected image. This avoids severe stretching or compression in the height direction of the corrected image, effectively reducing the probability of distortion in the image content and ensuring the accuracy of subsequent image processing.

[0144] In an optional embodiment, when the corner point determination module 1302 determines the second corner point corresponding to each first corner point, it is used to:

[0145] The greater of the lengths of the first line segment and the second line segment is determined as the width of the rectangle;

[0146] The height of the rectangle is obtained by multiplying the ratio of the larger length to the smaller length by the target height.

[0147] The second corner point corresponding to each first corner point is determined based on the width and height of the rectangle.

[0148] In an optional embodiment, when the corner point determination module 1302 determines the second corner point corresponding to each first corner point, it is used to:

[0149] The width of the rectangle is determined based on the computing resource utilization rate; the width of the rectangle is negatively correlated with the computing resource utilization rate.

[0150] The height of the rectangle is determined based on its width and its aspect ratio.

[0151] The second corner point corresponding to each first corner point is determined based on the width and height of the rectangle.

[0152] In an optional embodiment, when the corner point determination module 1302 determines the second corner point corresponding to each first corner point, it is used to:

[0153] Obtain the pre-configured width of the rectangle;

[0154] The height of the rectangle is determined based on its width and its aspect ratio.

[0155] The second corner point corresponding to each first corner point is determined based on the width and height of the rectangle.

[0156] In an optional embodiment, determining the second corner point corresponding to each first corner point includes:

[0157] Multiple image examples are determined based on the ratio of the target height to the target width, and the aspect ratio of each image example is the ratio of the target height to the target width; the widths of different image examples are different.

[0158] Showing examples of various images;

[0159] If a selection is obtained for any image example, the four vertices of the image example are determined as the four second corner points corresponding to the four first corner points.

[0160] In an optional embodiment, the target image is an image of a book; when the corner detection module 1301 performs corner detection on the target image, it is used to:

[0161] Corner detection is performed on the complete page in the target image to obtain the four first corner points of the complete page.

[0162] This application also provides an electronic device in its embodiments. (See reference...) Figure 14 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic device in these embodiments can be a terminal device (e.g., an in-vehicle system, a large-screen device, a smart home device, a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) or a server (which can be a single server, a server cluster, or a cloud server, etc.). Figure 14 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0163] like Figure 14 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage device 1408 into a random access memory (RAM) 1403. When the electronic device is powered on, the RAM 1403 also stores various programs and data required for the operation of the electronic device. The processing unit 1401, ROM 1402, and RAM 1403 are interconnected via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0164] Typically, the following devices can be connected to I / O interface 1405: input devices 1406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1408 including, for example, memory cards, hard drives, etc.; and communication devices 1409. Communication device 1409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 14 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0165] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the image correction methods provided in this application.

[0166] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the image correction methods provided in this application.

[0167] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0169] In the above embodiments, the functionality can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented entirely or partially as a computer program product. Those skilled in the art can use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.

[0170] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0171] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0172] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image correction method, characterized in that, include: Corner detection is performed on the target image to obtain the four first corner points of the target object in the target image; Determine the second corner point corresponding to each first corner point, where the aspect ratio of the rectangle with the four second corner points as vertices is the ratio of the target height to the target width; The target width is the smaller of the length of the first line segment between the top left and top right corner points and the length of the second line segment between the bottom left and bottom right corner points; the target height is greater than or equal to the minimum of the first and second vertical distances, and less than or equal to the average of the first and second vertical distances; the first vertical distance is the vertical distance from the top left corner point to the second line segment, and the second vertical distance is the vertical distance from the top right corner point to the second line segment; The target object is corrected based on the mapping relationship from the four first corner points to the four second corner points to obtain the corrected image.

2. The method according to claim 1, characterized in that, Determining the second corner point corresponding to each first corner point includes: The greater of the lengths of the first line segment and the second line segment is determined as the width of the rectangle; The height of the rectangle is obtained by multiplying the ratio of the larger length to the smaller length by the target height. The second corner point corresponding to each first corner point is determined based on the width and height of the rectangle.

3. The method according to claim 1, characterized in that, Determining the second corner point corresponding to each first corner point includes: The width of the rectangle is determined based on the computing resource utilization rate; the width of the rectangle is negatively correlated with the computing resource utilization rate. The height of the rectangle is determined based on its width and its aspect ratio. The second corner point corresponding to each first corner point is determined based on the width and height of the rectangle.

4. The method according to claim 1, characterized in that, Determining the second corner point corresponding to each first corner point includes: Obtain the pre-configured width of the rectangle; The height of the rectangle is determined based on its width and its aspect ratio. The second corner point corresponding to each first corner point is determined based on the width and height of the rectangle.

5. The method according to claim 1, characterized in that, Determining the second corner point corresponding to each first corner point includes: Multiple image examples are determined based on the ratio of the target height to the target width, and the aspect ratio of each image example is the ratio of the target height to the target width; the widths of different image examples are different. Showing examples of various images; If a selection is obtained for any image example, the four vertices of the image example are determined as the four second corner points corresponding to the four first corner points.

6. The method according to claim 1, characterized in that, The target image is an image of a book; the corner detection of the target image includes: Corner detection is performed on the complete page in the target image to obtain the four first corner points of the complete page.

7. An image correction device, characterized in that, include: The corner detection module is used to perform corner detection on the target image to obtain the four first corners of the target object in the target image; The corner point determination module is used to determine the second corner point corresponding to each first corner point. The aspect ratio of the rectangle with the four second corner points as vertices is the ratio of the target height to the target width. The target width is the smaller of the length of the first line segment between the top left and top right corner points and the length of the second line segment between the bottom left and bottom right corner points; the target height is greater than or equal to the minimum of the first and second vertical distances, and less than or equal to the average of the first and second vertical distances; the first vertical distance is the vertical distance from the top left corner point to the second line segment, and the second vertical distance is the vertical distance from the top right corner point to the second line segment; The correction module is used to correct the target object based on the mapping relationship from the four first corner points to the four second corner points, so as to obtain the corrected image.

8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the image correction method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, The electronic device includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the image correction method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the image correction method as described in any one of claims 1 to 6.

Images