Image correction method, image correction device, and program

Abstract

This invention provides a method for correcting image distortion caused by the shooting angle and folds. [Solution] The image correction method implemented by a computer includes acquiring a captured image generated by photographing a rectangular piece of paper containing one or more folds. The image correction method further includes approximating the contour of the target portion of the paper in the captured image with a polygon. The polygon includes five or more observation points observed on the contour as vertices. The image correction method further includes dividing the region enclosed by the polygon into two or more quadrilaterals based on the five or more observation points, generating two or more congruent rectangles by projective transformation of the two or more quadrilaterals, and generating a corrected image containing two or more congruent rectangles.

Description

[Technical Field]

[0001] This disclosure relates to an image correction method, an image correction apparatus, and a program. [Background technology]

[0002] One known method for converting paper documents into digital format is to photograph the paper documents using a camera or smartphone to generate image data. In this method, distortion may occur in the portion of the image that shows the paper document (hereinafter referred to as the "target portion") due to factors such as the shooting angle. Techniques have been proposed to correct image distortion in order to make information such as text contained in the target portion of the image easier to read.

[0003] Regarding techniques for correcting image distortion, Japanese Patent Publication No. 2020-102235 (Patent Document 1) discloses a program that causes a mobile terminal to perform a correction process to generate a corrected rectangular image by trapezoidal correction of a specified trapezoidal shape in an image, and an output process to output the corrected image data. The technology of Patent Document 1 makes it possible to project and transform a target portion that is distorted into a trapezoid due to the shooting angle into a rectangle, which is the shape of paper media. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-102235 [Overview of the project] [Problems that the invention aims to solve]

[0005] In the field of distance learning, answer sheets, which are paper documents, are exchanged between the distance learning provider and the learner. Traditionally, learners would fill out the answer sheets mailed to them by the distance learning provider and return them to the provider. In recent years, with the spread of the internet, learners can take pictures of their answer sheets and send the resulting image data to the distance learning provider.

[0006] Answer sheets may be folded evenly (e.g., into thirds) due to the size of the envelope used for mailing or for the convenience of the learner's management of the answer sheets. Therefore, in addition to distortion caused by the angle of photography, distortion caused by the folds may occur in the part of the image where the answer sheet is visible. When the technology described in Patent Document 1 is applied to an image containing distortion caused by folds, the technology treats the part where the answer sheet is visible as the target area, and treats the target area as a trapezoid and corrects it into a rectangle. Therefore, the corrected image still contains distortion caused by folds, making it difficult for distance learning providers to read.

[0007] One objective of this disclosure is to correct image distortion caused by the shooting angle and folds in order to solve these problems. [Means for solving the problem]

[0008] One aspect of the present disclosure provides a computer-based image correction method comprising acquiring a captured image generated by photographing a rectangular paper medium containing one or more folds. The image correction method further comprises approximating the contour of the target portion of the paper medium in the captured image with a polygon. The polygon includes five or more observation points observed on the contour as its vertices. The image correction method further comprises dividing the region enclosed by the polygon into two or more quadrilaterals based on the five or more observation points. The image correction method further comprises generating two or more congruent rectangles by projective transformation of the two or more quadrilaterals, and generating a corrected image containing the two or more congruent rectangles.

[0009] One aspect of the present disclosure is an image correction device comprising an acquisition unit, an approximation unit, a division unit, and a generation unit. The acquisition unit acquires a captured image generated by photographing a rectangular paper medium containing one or more folds. The approximation unit approximates the contour of the target portion of the paper medium in the captured image as a polygon. The polygon includes five or more observation points observed on the contour as its vertices. The division unit divides the region enclosed by the polygon into two or more quadrilaterals based on the five or more observation points. The generation unit generates two or more congruent rectangles by projective transformation of the two or more quadrilaterals and generates a corrected image containing the two or more congruent rectangles.

[0010] Furthermore, a program relating to another aspect of this disclosure causes a computer to execute the above-described image correction method. [Effects of the Invention]

[0011] According to this disclosure, the image correction method, image correction apparatus, and program can correct image distortion caused by the shooting angle and folds. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows an example of the configuration of a distance learning system to which the image correction method according to the embodiment is applied. [Figure 2] This figure shows an example of a quadratic approximation. [Figure 3] This is a flowchart showing part of the image correction process. [Figure 4] This flowchart shows the remaining part of the image correction process. [Figure 5] This figure shows an example of the relationship between a captured image and a corrected image. [Figure 6] This figure shows an example of the correspondence between quadrilaterals and rectangles. [Figure 7] Figure 4 is a diagram illustrating step S13. [Figure 8] This diagram illustrates a second method for determining whether there is a contradiction in the boundary lines of two adjacent quadrilaterals. [Figure 9]This diagram shows five polygons that require two or more virtual points. [Figure 10] This figure shows an example of the movement of a virtual point. [Modes for carrying out the invention]

[0013] Embodiments according to this disclosure will be described below with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions of them will not be repeated. The embodiments described below may be selectively combined as appropriate.

[0014] The image correction method relating to this disclosure is applicable to systems in various fields that use images obtained by photographing rectangular paper media. Typically, the image correction method relating to this disclosure is applied to a correspondence education system in which correction work is performed using images obtained by photographing answer sheets. The answer sheet is an example of "paper media" relating to this disclosure. Below, an example of applying the image correction method relating to this disclosure to a correspondence education system is described. However, the systems to which the image correction method relating to this disclosure is applied are not limited to correspondence education systems. The image correction method relating to this disclosure is applicable to systems that photograph paper media through digitalization and perform digital processing. An example of such a system is a system that transmits images obtained by photographing a rectangular application form sent by mail. In this case, the application form is an example of "paper media" relating to this disclosure.

[0015] <Configuration of the distance learning system> Figure 1 is a diagram showing an example of the configuration of a distance learning system to which the image correction method according to the embodiment is applied. The distance learning system shown in Figure 1 includes terminals 10 and 30. Terminal 10 generates a captured image 20 by taking a picture of the answer sheet 40 in response to an operation from the learner. Terminal 10 corrects the captured image 20 and generates a corrected image R2. The corrected image R2 is rectangular and has the same aspect ratio as the answer sheet 40. The size of the corrected image R2 is predetermined. The specific method for correcting the captured image 20 will be described later. Terminal 10 transmits the corrected image R2 to terminal 30. Terminals 10 and 30 include tablets, smartphones, personal computers, or other information processing devices.

[0016] As described above, the answer sheet 40 may contain one or more folds that were made when it was sent by the correspondence education provider or when it was managed by the learner. Typically, the answer sheet 40 contains one or more folds that divide it into multiple equal rectangular areas. Therefore, the terminal 10 performs a correction on the captured image 20 that takes into account that the answer sheet 40 may contain one or more folds that divide it into multiple equal rectangular areas. Note that "equal" is not limited to perfectly equal area and aspect ratio, but also includes approximately equal areas where the variation in area and aspect ratio is 5% or less.

[0017] Terminal 10 comprises a camera 11, a processor 12, memory 13, storage 14, a communication interface 15, and a user interface 16. Terminal 10 is an example of the “image correction device” of this disclosure.

[0018] Camera 11 uses an image sensor such as a CCD (Charged-Coupled Device), MOS (Metal-oxide-semiconductor), or CMOS (Complementary Metal-Oxide-Semiconductor) to capture objects in the field of view. Camera 11 is a two-dimensional camera. Camera 11 may be built into the terminal 10 or attached externally to the terminal 10. Camera 11 captures the field of view in which the answer sheet 40 is located, in response to the learner's operation. Camera 11 outputs the captured image 20 obtained from the capture.

[0019] The processor 12 includes a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), etc. The processor 12 reads the program 141 stored in the storage 14 and expands it into the memory 13. The processor 12 executes the expanded program 141. By the processor 12 executing the program 141, the acquisition unit 121, approximation unit 122, division unit 123, and generation unit 124 are realized.

[0020] Memory 13 includes, for example, a volatile storage device such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory).

[0021] Storage 14 includes, for example, a non-volatile storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or flash memory. Storage 14 also includes a storage device located inside the terminal 10 and online storage. Storage 14 stores program 141 and various information used to execute program 141. Storage 14 may also store one or more captured images 20 in an image folder 142. Program 141 is downloaded from an app store (not shown). Program 141 includes a set of commands for acquiring captured images 20, correcting captured images 20, and outputting a corrected image R2.

[0022] The communication interface 15 communicates with the terminal 30 via a wired or wireless network.

[0023] The user interface 16 provides information to the user and accepts input from the user. The user interface 16 includes, for example, a display and input devices. The input devices include, for example, a mouse, a keyboard, and a touchpad. The user interface 16 may also be configured as a touch panel in which the touchpad and display are integrated.

[0024] The acquisition unit 121 acquires the captured image 20 generated by photographing the rectangular answer sheet 40 from the camera. As described above, the answer sheet 40 may include one or more folds that divide the answer sheet 40 into multiple equal rectangular areas. The acquisition unit 121 may also acquire the captured image 20 to be corrected from the image folder 142 in response to an input instruction.

[0025] The approximation unit 122 approximates the contour of the area in the captured image 20 where the answer sheet 40 is visible as a polygon. The polygon includes the observation points observed on the contour as vertices.

[0026] Specifically, the approximation unit 122 performs edge extraction on the captured image 20 and polygonal approximation on the extracted edges. As an example of edge extraction, the approximation unit 122 performs image processing using a known algorithm such as the Canny algorithm. Alternatively, the processor 12 may perform image processing using a differential filter such as a Sobel filter as another edge extraction method. As an example of polygonal approximation, the approximation unit 122 performs first-order and second-order approximation.

[0027] In the first-order approximation, the approximation unit 122 approximates the extracted edges to one or more polygons by performing image processing using a known algorithm such as the Douglas-Peucker algorithm. In addition to the contour of the target area, the approximation unit 122 can recognize characters and borders inside the target area as edges. The approximation unit 122 determines that the polygon with the largest area among the one or more approximated polygons, or the outer polygon, is the polygon corresponding to the target area. The polygon generated by the first-order approximation includes three or more candidate observation points as vertices. The approximation unit 122 may also perform edge extraction and first-order approximation using the OpenCV® library in the Python language.

[0028] Figure 2 shows an example of a quadratic approximation. Figure 2 shows candidate observation points pc100, pc101, and pc102 observed on the contour, and the edges pc100pc101 and pc101pc102. In the quadratic approximation, the approximation unit 122 compares the difference between the inferior angle (the smaller of the angles shared by the two edges) between two adjacent edges and 180 degrees and a threshold for each of the three or more candidate observation points. Specifically, the threshold is set to a value of 15 degrees or less. The approximation unit 122 deletes the corresponding candidate observation point depending on whether the difference is less than or equal to the threshold. For example, if the approximation unit 122 determines that the inferior angle between edges pc100pc101 and pc101pc102 is less than or equal to the threshold, it deletes candidate observation point pc101. The approximation unit 122 determines the remaining candidate observation points as observation points after performing a quadratic approximation on each of the three or more candidate observation points. In this way, the approximation unit 122 approximates the contour of the target portion as a polygon that includes the observation points observed on the contour as vertices.

[0029] Furthermore, when performing a quadratic approximation, the approximation unit 122 may calculate two vectors from two adjacent edges and compare the angle between the two vectors with a threshold. Specifically, the processor 12 calculates vector Va for one of the two adjacent edges, starting from the point of intersection. The processor 12 also calculates vector Vb for the other of the two adjacent edges, ending from the point of intersection. The processor 12 can then calculate the angle θ between the two vectors according to the following equation, which uses the dot product of the two vectors and the inverse cosine function. θ = arccos(Va·Vb / |Va|·|Vb|) In this case, the approximation part 122 removes candidate observation points that are intersection points of two edges, for example, depending on whether -(π / 12)≦θ≦π / 12 is satisfied.

[0030] The division unit 123 divides the region enclosed by the polygon into one or more quadrilaterals based on the arrangement of four or more observation points included in the polygon. Specifically, the division unit 123 divides the region enclosed by the polygon into two or more quadrilaterals depending on whether the polygon contains five or more observation points. If the polygon contains only four observation points, the division unit 123 considers the region enclosed by the polygon as a single quadrilateral. The specific division method will be described later.

[0031] The generation unit 124 generates a corrected image R2. Depending on whether the polygon contains only four observation points, the generation unit 124 generates the corrected image R2 by projecting the region enclosed by the polygon (quadrilateral) into a rectangle. Depending on whether the polygon contains five or more observation points, the generation unit 124 generates two or more congruent rectangles by projecting the two or more quadrilaterals divided by the division unit 123, and generates a corrected image R2 containing two or more congruent rectangles. The specific method for generating the corrected image R2 will be described later.

[0032] <Image Correction Method> The image correction method of this disclosure will be explained with reference to Figures 3 to 10. Figure 3 is a flowchart showing part of the image correction flow. Figure 4 is a flowchart showing the remaining part of the image correction flow. The processes shown in Figures 3 and 4 are performed in response to the input of instructions to submit the answer sheet. Figure 5 is a diagram showing an example of the relationship between the captured image and the corrected image.

[0033] In the first step S1, the processor 12, acting as the acquisition unit 121, acquires the captured image 20. For example, the processor 12 activates the camera 11 and outputs a screen prompting the user to perform a shooting operation to the user interface 16. Upon receiving the shooting operation, the processor 12 outputs a shooting command to the camera 11 and acquires the captured image 20 from the camera 11. Alternatively, the processor 12 may acquire the captured image 20 from the image folder 142 in response to input to the user interface 16. After step S1, the processor 12 moves the process to step S2.

[0034] In step S2, the processor 12, acting as the approximation unit 122, approximates the contour of the target area R1 on which the answer sheet 40 is visible as a polygon. The shape of the polygon is determined by the shape of the target area R1 and one or more functions used for polygon approximation. One or more functions include, for example, the cv2.findContours() function from the OpenCV® library. In the example shown in Figure 5, the target area R1 is approximated as a polygon P110. The polygon P110 includes observation points p1, p2, p3, p4, p5, and p6 as vertices. After step S2, the processor 12 moves the processing to step S3.

[0035] In step S3, the processor 12, which operates as the division unit 123, determines whether the number of observation points included in the polygon is 4 or more. If the processor 12 determines that the number of observation points included in the polygon is 3 or less (NO in step S3), it outputs an error (step S4). After step S4, the processor 12 terminates processing. If the processor 12 determines that the number of observation points included in the polygon is 3 or less (YES in step S3), it moves the process to step S5.

[0036] In step S5, the processor 12, which operates as the division unit 123, assigns attributes to four or more observation points included in the polygon. The attributes include a first attribute corresponding to the corners of the answer sheet 40 and a second attribute corresponding to the intersection of the fold of the answer sheet 40 and the edge of the answer sheet 40. For example, the processor 12 assigns the first attribute to an observation point when the absolute value of the difference between the inferior angle formed by two adjacent sides and 90 degrees is less than or equal to a threshold. The processor 12 assigns the second attribute to an observation point when the absolute value of the difference between the inferior angle formed by two adjacent sides and 90 degrees exceeds a threshold. The processor 12 may also calculate the angle formed by two adjacent sides using vectors, similar to the quadratic approximation. Specifically, the processor 12 calculates two vectors from the two adjacent sides in the same manner as the quadratic approximation. Processor 12 calculates the angle between two sides using the dot product of two vectors and the inverse cosine function. The threshold is set to a value less than 45 degrees (e.g., 30 degrees). In the example shown in Figure 5, processor 12 assigns a first attribute to observation points p1, p3, p4, and p6, and a second attribute to observation points p2 and p5. After step S5, processor 12 moves the process to step S6.

[0037] In step S6, the processor 12 determines whether the number of observation points corresponding to the corners (observation points assigned the first attribute) is 4. If the processor 12 determines that the number of observation points with the first attribute is not 4 (NO in step S6), it moves the process to step S4. That is, the processor 12 outputs an error and terminates the process. If the processor 12 determines that the number of observation points corresponding to the corners is 4 (YES in step S6), it moves the process to step S7.

[0038] In step S7, the processor 12 classifies the four observation points corresponding to the angles into one of the first, second, third, or fourth angles. For example, the processor 12 classifies the observation point closest to the origin among the four observation points assigned the first attribute into the first angle. Next, the processor 12 classifies the three observation points assigned the first attribute that appear when tracing the sides of the polygon counterclockwise from the observation point of the first angle into the second, third, and fourth angles, respectively. In the example shown in Figure 5, the processor 12 classifies the observation points p1, p3, p4, and p6 belonging to the first attribute into the first, second, third, and fourth angles, respectively. After step S7, the processor 12 moves the processing to step S8.

[0039] In step S8, the processor 12 determines whether or not there is an observation point (an observation point to which the second attribute is assigned) that corresponds to the intersection of the fold of the answer sheet 40 and the edge of the answer sheet 40. If the processor 12 determines that there is no observation point with the second attribute (NO in step S8), it moves the process to step S9.

[0040] In step S9, each observation point is associated with a vertex of the rectangular frame that forms the outline of the corrected image R2 (step S9). Specifically, the processor 12 associates the top-left vertex, bottom-left vertex, bottom-right vertex, and top-right vertex of the rectangular frame that forms the outline of the corrected image R2 with the observation points belonging to the first, second, third, and fourth corners, respectively. After step S9, the processor 12 moves the processing to step S10.

[0041] Figure 6 shows an example of the correspondence between each observation point and the vertices of the rectangular frame. Figure 6 shows polygon P120 and corrected image R2. Figure 6 shows an example where YES is determined in step S6 and NO is determined in step S8. The processor 12 determines that polygon P120 is a quadrilateral based on the absence of observation points included in the second attribute. As described above, the size of corrected image R2 is predetermined according to the size of the answer sheet 40. The horizontal width of corrected image R2 is Xmax, and the vertical height of corrected image R2 is Ymax. For example, when the size of the answer sheet 40 is A4 size, Xmax:Ymax is set to 210:297. The rectangular frame R200, which is the outline of corrected image R2, includes vertices C1, C2, C3, and C4 in a counterclockwise direction from the top left corner. The coordinates of vertices C1, C2, C3, and C4 are represented as (0,0), (0,Ymax), (Xmax,Ymax), and (Xmax,0), respectively. Polygon P120 (quadrilateral) contains observation points p11, p12, p13, and p14 as vertices, corresponding to the first, second, third, and fourth angles, respectively. Processor 12 associates observation points p11, p12, p13, and p14 with vertices C1, C2, C3, and C4, respectively.

[0042] In step S10, the processor 12, acting as the generation unit 124, calculates one or more projection transformation matrices that project one or more quadrilaterals onto their corresponding rectangles. In the example shown in Figure 6, the processor 12 calculates a projection transformation matrix that projects polygon P120 onto rectangular frame R200. After step S10, the processor 12 moves the processing to step S11.

[0043] The projection transformation from a quadrilateral to a rectangle is expressed by the following equation 1. Equation 1 includes nine coefficients a, b, c, d, e, f, g, h, and s, as well as the X coordinates xi and Y coordinates yi of the source and the X coordinates Xi and Y coordinates Yi of the transformed object.

[0044]

number

[0045] The nine coefficients in Equation 1 are calculated using the coordinates of four observation points and the coordinates of the corresponding rectangle vertices. Specifically, Equation 1 is transformed into Equations 2, 3, and 4. Processor 12 derives four equations by substituting the X coordinates of the four observation points and the corresponding X coordinates of the four vertices into Equation 3. Similarly, Processor 12 derives four equations by substituting the Y coordinates of the four observation points and the corresponding Y coordinates of the four vertices into Equation 4. That is, Processor 12 uses Equations 3 and 4 to derive eight equations containing the undetermined constants a, b, c, d, e, f, g, and h. Processor 12 uses these eight equations containing the eight undetermined constants to calculate the eight undetermined constants. Processor 12 also calculates the value of the undetermined constant s by substituting the calculated coefficients g and h, along with the X and Y coordinates of the observation points, into Equation 2. For example, in the example shown in Figure 6, processor 12 calculates the undetermined constants using eight equations derived from the XY coordinates of observation points p11, p12, p13, and p14, and the corresponding XY coordinates of four vertices C1, C2, C3, and C4. Alternatively, processor 12 may use the cp2.getPerspectipeTransform() function from the OpenCP® library to determine the specific projection transformation matrix.

[0046] In step S11, the processor 12 performs a projection transformation on one or more quadrilaterals to generate a corrected image. That is, the processor 12 performs a projection transformation on one or more quadrilaterals using the corresponding projection transformation matrix. In the example shown in Figure 6, the processor 12 generates a corrected image R2 by projecting the region (quadrilateral) enclosed by polygon P120 into a rectangle of the same shape and size as the rectangular frame R200. For example, the processor 12 uses the cp2.warpPerspectipe() function from the OpenCP® library. However, the method of performing the projection transformation is not limited to this. After step S11, the processor 12 terminates processing.

[0047] As shown in Figures 3 and 4, when the processor 12, which operates as the division unit 123, determines that there is an observation point having a second attribute (YES in step S8), it moves the process to step S12. Note that if YES is obtained in step S8, the polygon approximated in step S2 will have five or more observation points as vertices.

[0048] In step S12, the processor 12 calculates the number of divisions N in the first direction and the number of divisions M in the second direction. The first direction is the direction parallel to the line segment whose endpoints are the two observation points classified as the second and third angles, or the line segment whose endpoints are the two observation points classified as the first and fourth angles. The second direction is the direction that intersects the first direction (typically, the direction perpendicular to the first direction). Specifically, the second direction is the direction parallel to the line segment whose endpoints are the two observation points classified as the first and second angles, or the line segment whose endpoints are the two observation points classified as the third and fourth angles. Of the four observation points assigned the first attribute, the observation point closest to the origin is classified as the first angle, and the remaining three observation points that appear when tracing the polygonal line segment counterclockwise from the observation point of the first angle are classified as the second, third, and fourth angles in order. Therefore, the first direction is roughly parallel to the horizontal direction of the captured image 20. Therefore, the number of divisions N in the first direction corresponds to the number of horizontal regions divided by the folds in the answer sheet 40. Similarly, the number of divisions M in the second direction corresponds to the number of vertical regions divided by the folds in the answer sheet 40.

[0049] In step S12, processor 12 first classifies each observation point having a second attribute into one of the first to fourth groups. Observation points belonging to the same group correspond to folds on the same side of the answer sheet 40. Processor 12 classifies observation points that appear between the first and second angles when tracing counterclockwise along the sides of the polygon into the first group. Processor 12 classifies observation points that appear between the second and third angles when tracing counterclockwise along the sides of the polygon into the second group. Processor 12 classifies observation points that appear between the third and fourth angles when tracing counterclockwise along the sides of the polygon into the third group. Processor 12 classifies observation points that appear between the fourth and first angles when tracing counterclockwise along the sides of the polygon into the fourth group. In the example shown in Figure 5, processor 12 classifies observation point p2 into the first group and observation point p5 into the third group.

[0050] In step S12, processor 12 determines the division number N to be the larger of the number of observation points belonging to the second group and the number of observation points belonging to the fourth group, plus 1. Processor 12 also determines the division number M to be the larger of the number of observation points belonging to the first group and the number of observation points belonging to the fourth group, plus 1. In the example shown in Figure 5, processor 12 determines that the division number N is 1 and the division number M is 2. After step S12, processor 12 moves the process to step S13.

[0051] In step S13, the processor 12 associates each of the five or more observation points with one of the vertices of a grid of N × M congruent rectangles. After step S13, the processor 12 moves the processing to step S14.

[0052] The details of steps S12 and S13 will be described with reference to FIG. 5 and FIG. 7. FIG. 7 is a diagram for explaining step S13 shown in FIG. 4. The processor 12 divides the rectangular frame R200, which is the contour of the corrected image R2, into N×M congruent rectangles arranged in a grid pattern with N pieces in the horizontal direction and M pieces in the vertical direction. Hereinafter, among the N×M rectangles, the j-th rectangle from the left in the horizontal direction and the k-th rectangle from the top in the vertical direction is referred to as rectangle R j,k . j is an integer from 1 to N. k is an integer from 1 to M. The N×M rectangles are defined by vertices arranged in N+1 pieces in the horizontal direction and M+1 columns in the vertical direction. Hereinafter, the vertex in the n-th column from the left and the m-th row in the downward direction is referred to as vertex v n,m . n is an integer from 0 to N. m is an integer from 0 to M. The four vertices of rectangle R j,k include vertices v j-1,k-1 , v j-1,k , v j,k , v j,k-1 . Note that vertices v 0,0 , v 0,M , v N,M , v N,0 respectively coincide with the vertices C1, C2, C3, C4 of the rectangular frame R200. As described above, the coordinates of vertices C1, C2, C3, C4 are expressed as (0,0), (0, Ymax), (Xmax, Ymax), (Xmax, 0), respectively. Therefore, the coordinates of vertex v n,m are expressed as (n×Xmax / N, m×Ymax / M).

[0053] In the example shown in FIG. 5, the division number N is 1, and the division number M is 2. Therefore, the rectangular frame R200 is divided into two congruent rectangles R 1,1 , R 1,2 arranged side by side in the vertical direction. Rectangle R 1,1 includes vertices v 0,0 , v 0,1 , v 1,1 , v 1,0 . Rectangle R 1,2 includes vertices v 0,1 , v 0,2 , v 1,2 , v 1,1 . Note that vertices v 0,0 , v 0,2 , v 1,2 , v1,0 The coordinates of vertices v coincide with the coordinates of vertices C1, C2, C3, and C4, and are represented as (0,0), (0,Ymax), (Xmax,Ymax), and (Xmax,0), respectively. 0,1 ,v 1,1 The coordinates of these points are represented as (0, Ymax / 2) and (Xmax, Ymax / 2), respectively.

[0054] In the example shown in Figure 7, the number of divisions N is 1 and the number of divisions M is 3. Therefore, the rectangular frame R200 is a congruent rectangle R with three rectangles arranged vertically. 1,1 ,R 1,2 ,R 1,3 It is divided into rectangles R 1,1 is vertex v 0,0 ,v 0,1 ,v 1,1 ,v 1,0 Includes. Rectangle R 1,2 is vertex v 0,1 ,v 0,2 ,v 1,2 ,v 1,1 Includes. Rectangle R 1,3 is vertex v 0,2 ,v 0,3 ,v 1,3 ,v 1,2 This includes vertex v 0,0 ,v 0,3 ,v 1,3 ,v 1,0 These correspond to vertices C1, C2, C3, and C4 of the rectangular frame R200, respectively, and are represented as (0,0), (0,Ymax), (Xmax,Ymax), and (Xmax,0). Vertex v 0,1 ,v 1,1 ,v 0,2 ,v 1,2 The coordinates are represented as (0, Ymax / 3), (Xmax, Ymax / 3), (0, 2Ymax / 3), and (Xmax, 2Ymax / 3), respectively.

[0055] Processor 12 classifies four observation points from the five or more observation points into the first, second, third, and fourth angles, and places them at vertex v 0,0 ,v 0,M ,v N,M ,v N,0Each is associated with a specific point. In the example shown in Figure 5, observation points p1, p3, p4, and p6 are classified as the first, second, third, and fourth angles, respectively. Therefore, the processor 12 assigns observation points p1, p3, p4, and p6 to vertex v 0,0 ,v 0,M ,v N,M ,v N,0 Each is associated with a specific point. In the example shown in Figure 7, observation points p21, p24, p25, and p27 are classified as the first, second, third, and fourth angles, respectively. Therefore, the processor 12 assigns observation points p21, p24, p25, and p27 to vertex v 0,0 ,v 0,3 ,v 1,3 ,v 1,0 Assign a correspondence to each.

[0056] Next, the processor 12 associates the remaining observation points (i.e., observation points assigned the second attribute) among the five or more observation points with one of the vertices. Specifically, if the number of observation points classified into the second group is equal to the number of observation points classified into the fourth group, the processor 12 associates the observation points classified into the second and fourth groups with the vertices of the rectangle according to the first association method. The first association method associates the observation point classified into the second group that is the i-th closest to the second corner with vertex v i,M Corresponding to this, the observation point closest to the first corner among the observation points classified into the fourth group is vertex v i,0 This is a method of associating the observation points with the vertices of the rectangle. Similarly, if the number of observation points classified into the first group is equal to the number of observation points classified into the third group, the processor 12 associates the observation points classified into the first and third groups with the vertices of the rectangle according to the second association method. The second association method assigns the observation point that is the i-th closest to the first corner among the observation points classified into the first group to vertex v 0,i By associating them, the observation point closest to the fourth corner among the observation points classified into the third group is vertex v. n,i This is a method of creating a correspondence.

[0057] In the example shown in Figure 5, polygon P110 includes observation point p2 belonging to the first group and observation point p5 belonging to the third group. Since the number of observation points belonging to the first group is the same as the number of observation points belonging to the third group, processor 12 assigns observation point p2 belonging to the first group to vertex v 0,1 Furthermore, the processor 12 associates observation point p5, which belongs to the third group, with vertex v. 1,1 To associate with something.

[0058] If the number of observation points classified into the second group differs from the number of observation points classified into the fourth group, the processor 12 associates the observation points belonging to the group with the larger number of observation points (the second group or the fourth group) with the vertices of the rectangle, according to the first matching method described above. The processor 12 selects a vertex to associate with an observation point, depending on the position of the observation point belonging to the group with the smaller number of observation points (the second group or the fourth group). Specifically, if the number of observation points in the second group (an integer greater than or equal to 1) is less than the number of observation points belonging to the fourth group, the processor 12 calculates a first sum of the lengths of the edges obtained by sequentially connecting the observation points from the second corner to the third corner. Next, the processor 12 calculates a second sum of the lengths of the edges obtained by sequentially connecting the observation points from the second corner to the observation point of interest belonging to the second group. The processor 12 identifies an integer i from 1 to N that is closest to the product of the quotient obtained by dividing the second sum by the first sum and the number of divisions N. Then, the processor 12 assigns the observation point of interest to vertex v i,M It maps to vertex v. i,0 To associate with something.

[0059] If the number of observation points classified into the first group differs from the number of observation points classified into the third group, the processor 12 associates the observation points belonging to the group with the larger number of observation points with the vertices of the rectangle, according to the second matching method described above. The processor 12 selects a vertex to associate with an observation point, depending on the position of the observation point belonging to the group with the smaller number of observation points. Specifically, if the number of observation points in the first group (an integer greater than or equal to 1) is less than the number of observation points belonging to the third group, the processor 12 calculates a first sum of the lengths of the edges obtained by sequentially connecting the observation points from the first corner to the second corner. Next, the processor 12 calculates a second sum of the lengths of the edges obtained by sequentially connecting the observation points from the first corner to the observation point of interest belonging to the first group. From among integers from 1 to M, the integer i closest to the product of the quotient obtained by dividing the second sum by the first sum and the number of divisions M is identified. The observation point of interest is assigned to vertex v 0,i It maps to vertex v. i,n To associate with something.

[0060] In the example shown in Figure 7, polygon P130 includes observation points p22 and p23 belonging to the first group, and observation point p26 belonging to the third group. The number of observation points belonging to the first group is greater than the number of observation points belonging to the third group. Therefore, the processor 12 assigns observation point p22, which is close to the first corner (observation point p21), to vertex v 0,1 By associating them, the observation point p23, which is farther from the first corner (observation point p21), is set to vertex v 0,2The processor 12 then maps the observation point p26, which belongs to the third group, to vertex v. 1,1 To associate with something.

[0061] In step S14, the processor 12 determines whether there are insufficient observation points for the vertices of the N × M rectangles. Specifically, the processor 12 compares (N+1) × (M+1) with the number of observation points. If (N+1) × (M+1) is greater than the number of observation points, the processor 12 determines that there are insufficient observation points. If (N+1) × (M+1) is equal to the number of observation points, the processor 12 determines that there are no insufficient observation points. When it is determined that there are insufficient observation points (YES in step S14), the processor 12 moves the process to step S17.

[0062] In the example shown in Figure 5, the processor 12 identifies six observation points p1, p2, p3, p4, p5, and p6 from the captured image 20. The processor 12 then determines that the number of divisions N is 1 and the number of divisions M is 2. Since (N+1)×(M+1) matches the number of observation points, the processor 12 determines that there are no shortage of observation points. When it is determined that there are no shortage of observation points (NO in step S14), the processor 12 moves the process to step S15.

[0063] In step S15, the processor 12 operating as the dividing unit 123 divides the region enclosed by the polygon into N×M quadrangles. The processor 12 defines N×M quadrangles each including, as vertices, four observation points corresponding to the four vertices contained in each of the N×M rectangles. That is, the processor 12 sets the j-th in the first direction and k-th in the second quadrangle Q j,k to have four observation points corresponding to the four vertices of the j-th in the horizontal direction and k-th in the vertical direction rectangle R j,k as vertices, and divides the region enclosed by the polygon into N×M quadrangles. In the example shown in FIG. 5, the processor 12 defines a quadrangle Q 1,1 that takes the vertex v 0,0 , v 0,1 , v 1,1 , v 1,0 of the rectangle R and includes the corresponding observation points p1, p2, p5, p6 as vertices 1,1 , and a quadrangle Q 1,2 that takes the vertex v 0,1 , v 0,2 , v 2,2 , v 1,1 of the rectangle R and includes the corresponding observation points p2, p3, p4, p5 as vertices 1,2 . After step S15, the processor 12 shifts the process to step S16.

[0064] In step S16, the processor 12 operating as the dividing unit 123 determines whether there is a contradiction between two post-projective-transformation boundary lines generated by projectively transforming the boundary line of two adjacent quadrangles using the two projective transformation matrices corresponding to the two quadrangles respectively. Specifically, the processor 12 determines whether there is a contradiction between the two post-projective-transformation boundary lines by one of the following two determination methods.

[0065] In the first determination method, the processor 12 determines whether or not the coordinates of a point on the boundary line of two adjacent quadrilaterals become the same value through projective transformation for the two quadrilaterals. The processor 12 calculates two projective transformation matrices for projectively transforming the two adjacent quadrilaterals into respective corresponding rectangles. The projective transformation matrices are calculated using the same method as in step S10. The processor 12 calculates the difference between the two coordinates after projective transformation for the coordinates of a point (for example, a midpoint) on the boundary line, based on the two projective transformation matrices for the two quadrilaterals. When the coordinate difference is equal to or greater than a threshold, the processor 12 determines that there is an inconsistency between the two boundary lines after projective transformation. Note that the processor 12 may calculate the difference between the coordinates after projective transformation using the two projective transformation matrices for the respective coordinates of a plurality of points on the boundary line, and compare a representative value of the post-transformation differences (such as a maximum value, a minimum value, an average value, and a median value) with the threshold. In the example shown in Figure 5, the processor 12 processes the quadrilateral Q 1,1 into the rectangle R 1,1 and the projective transformation matrix for projective transformation, and the quadrilateral Q 1,2 into the rectangle R 1,2 to calculate respective projective transformation matrices for projective transformation. Next, the processor 12 calculates the difference between the two coordinates after projective transformation for the coordinates of the midpoint of the line segment p2p5. Then, the processor 12 determines whether or not there is an inconsistency between the two boundary lines after projective transformation according to the comparison result between the difference and the threshold.

[0066] In the second method of determination, processor 12 determines whether there is a contradiction in the boundary lines of two adjacent quadrilaterals based on the constraints of two projection transformation matrices corresponding to those quadrilaterals. As described above, processor 12 projects N×M quadrilaterals into N×M congruent rectangles. The projection transformation matrix for projecting quadrilaterals into rectangles is obtained by multiplying a first projection transformation matrix for projecting a quadrilateral into a square with side length 1, and a second projection transformation matrix for projecting a square with side length 1 into a rectangle. The second projection transformation matrix is ​​simply a matrix for scaling along the X and Y axes, and is the same for N×M quadrilaterals. Therefore, instead of the constraints of two projection transformation matrices corresponding to two adjacent quadrilaterals, the constraints of two first projection transformation matrices corresponding to those two quadrilaterals can be used. For this reason, the constraints of the two first projection transformation matrices are explained below.

[0067] Figure 8 illustrates a second method for determining whether there is a contradiction between two projected boundaries generated by projectively transforming the boundaries of two adjacent quadrilaterals using two first projection transformation matrices. Figure 8 shows the two quadrilaterals Q shown in Figure 5. 1,1 ,Q 1,2 This is shown. First, processor 12 processes two adjacent quadrilaterals Q 1,1 ,Q 1,2 The polygon P110 is rotated and translated such that one of the two vertices shared by it is located at the origin and the other is located on the positive X-axis. In Figure 8, points Tp1, Tp2, Tp3, Tp4, Tp5, and Tp6 are the points after the movement of the observation points p1, p2, p3, p4, p5, and p6, respectively. Quadrilateral TQ 1,1 This is quadrilateral Q 1,1 It corresponds to and includes points Tp1, Tp2, Tp5, and Tp6 as vertices. Quadrilateral TQ 1,2 This is quadrilateral Q 1,2 It corresponds to and includes points Tp2, Tp3, Tp4, and Tp5 as vertices.

[0068] Next, processor 12 moves the two quadrilaterals TQ after the move. 1,1 ,TQ 1,2We calculate two first projection transformation matrices that project the object onto two adjacent squares, respectively. The side length of the two squares is 1. The two squares share a vertex at the origin and on the X-axis. The two first projection transformation matrices are represented by equation 1 above and equation 5 below, respectively. Equation 5 is transformed into equations 6, 7, and 8. Equation 5 contains undetermined constants a', b', c', d', e', f', g', h', s', the X coordinate x'i and Y coordinate y'i of the object to be projected, and the X coordinate X'i and Y coordinate Y'i of the object after the projection transformation.

[0069]

number

[0070] Two quadrilaterals TQ 1,1 ,TQ 1,2 The point Tp2 on the origin is projected onto the origin (0,0), which is one of the common vertices of the two squares. Therefore, based on equations 3, 4, 7, and 8, 0=c=f=c'=f' holds true. Also, the two quadrilaterals TQ 1,1 ,TQ 1,2 The intersection points on the x-axis that constitute the boundary lines of the two squares are projected onto the intersection points on the x-axis that constitute the boundary lines of the two squares. Therefore, based on equations 4 and 8, for any xi, 0 = dxi / (gxi+1) = d'xi / (g'xi+1) holds. From this, we derive d = d' = 0. Thus, there exist undetermined constants a, b, e, g, h, a', b', e', g', and h'.

[0071] As mentioned above, the two quadrilaterals TQ 1,1 ,TQ 1,2 The intersection points on the X-axis that constitute the boundary of the two squares are projected onto the intersection points on the X-axis that constitute the boundary of the two squares. Therefore, based on equations 3 and 7, Xi = axi / (gxi+1) = a'xi / (g'xi+1) holds. Since axi / (gxi+1) = a'xi / (g'xi+1) always holds for any values ​​of Xi and xi, two equations a = a' and g = g' are derived. Also, processor 12 processes the quadrilateral TQ. 1,1 ,TQ 1,2By substituting the x-coordinate x0 of point Tp5, which is shared by the two squares, and the x-coordinate of vertex (1,0), which is shared by the two squares, into equation 3, the equation 1 = ax0 / (gx0 + 1) can be derived. Processor 12 then processes the two quadrilaterals TQ 1,1 ,TQ 1,2 From the four XY coordinates of points Tp1, Tp3, Tp4, and Tp6 that are not shared by the two sides, and the four XY coordinates of the corresponding vertices of the square, a total of eight equations can be derived, two each for equations 3, 5, 6, and 7. In this way, processor 12 can derive 11 equations that include at least one of the 10 undetermined constants a, b, e, g, h, a', b', e', g', h'. Based on 10 of these 11 equations, processor 12 determines the 10 undetermined constants. Next, processor 12 determines whether the difference between the left and right sides is less than or equal to a threshold when it substitutes the obtained constants into the equations that were not used to determine the undetermined constants (for example, a=a'). If processor 12 determines that the difference between the left and right sides is greater than the threshold, it determines that there is a contradiction in the boundary lines of the two quadrilaterals.

[0072] Using one of the two determination methods described above, the processor 12 determines whether there is a contradiction between the two projectively transformed boundaries of each pair of adjacent quadrilaterals. If the processor 12 determines that there is a contradiction between the two projectively transformed boundaries corresponding to any pair (YES in step S16), it moves the process to step S4. That is, the processor 12 outputs an error and terminates the process.

[0073] When it is determined that there is no inconsistency between the two projectively transformed boundaries for all pairs (NO in step S16), the processor 12, acting as the generation unit 124, performs the processing in steps S10 and S11. After that, the processor 12 terminates the process. In the example shown in Figure 5, polygon P110 is quadrilateral Q 1,1 ,Q 1,2 It is divided into quadrilateral Q. 1,1 The observation points p1, p2, p5, and p6, which are the vertices of the rectangle R, are located in the rectangle R. 1,1 Vertex v 0,0 ,v 0,1 ,v1,1 ,v 1,0 This is mapped to the quadrilateral Q. Therefore, the processor 12 maps these points to the quadrilateral Q. 1,1 Rectangle R 1,1 The projection transformation matrix is ​​calculated to perform the projection transformation on quadrilateral Q. Similarly, processor 12 calculates the projection transformation matrix for quadrilateral Q. 1,2 Rectangle R 1,2 The processor calculates a projection transformation matrix to project onto quadrilateral Q. Then, the processor 12 uses the calculated projection transformation matrix to project onto quadrilateral Q. 1,1 ,Q 1,2 The image is subjected to a projection transformation, and a corrected image R2 containing the two rectangles obtained by the projection transformation is generated.

[0074] In step S17, the processor 12 places one or more virtual points corresponding to one or more vertices among the N×M vertices of the rectangle that do not correspond to any observation points. That is, the processor 12 places virtual points for the vertices of the N×M rectangle according to the determination that there are insufficient observation points. The one or more virtual points are each placed at the starting point of their corresponding existence range. The processor 12 places virtual points for vertices v where no corresponding observation points exist. n,m A virtual point is defined for this. As mentioned above, n is an integer from 0 to N, and m is an integer from 0 to M. Also, when n is 0 or N, processor 12 defines vertex v n,m The range of existence of the virtual point corresponding to vertex v n,m-1 The starting point is the observation point or virtual point corresponding to it, and the vertex is v n,m+1 The line segment whose endpoint is the corresponding observation point or virtual point is on the line segment. Similarly, when m is 0 or M, processor 12 considers vertex v n,m The range of existence of the virtual point corresponding to vertex v n-1,m The starting point is the observation point or virtual point corresponding to it, and the vertex is v n+1,m It is defined as the line segment whose endpoint is the observation point or virtual point corresponding to . Also, when n is not 0 or N and m is not 0 or M, the processor 12 determines vertex v n,m The range of existence of the virtual point corresponding to vertex v n-1,m-1 ,v n-1,m ,v n-1,m+1 ,v n,m+1 ,v n+1,m+1 ,vn+1,m ,v n+1,m-1 ,v n,m-1 It is defined as an octagonal region whose vertices are the eight observation points or virtual points corresponding to each of the following. When n is not 0 or N and m is not 0 or M, the processor 12 determines the vertex v n,m The starting and ending points of the range of existence for the corresponding virtual point are defined as vertex v n-1,m-1 ,v n-1,m ,v n-1,m+1 ,v n,m+1 ,v n+1,m+1 ,v n+1,m ,v n+1,m-1 ,v n,m-1 Defined as an observation point or virtual point corresponding to any of the above. After step S17, the processor 12 moves the processing to step S18.

[0075] In the example shown in Figure 7, as described above, observation points p21, p22, p23, p24, p25, p26, and p27 are at vertex v 0,0 ,v 0,1 ,v 0,2 ,v 0,3 ,v 1,3 ,v 1,1 ,v 1,0 Each is associated with the corresponding vertex v. Therefore, processor 12 is assigned to vertex v. 1,2 It was determined that there was no corresponding observation point for vertex v 1,2 A virtual point vp1 corresponding to vertex v is placed within the captured image 20. Since the number of divisions N is 1 and the number of divisions M is 3, the processor 12 determines the range of existence of the virtual point vp1 as vertex v 1,1 Starting point p26, which corresponds to this observation point, vertex v 1,3 Let the corresponding observation point p25 be the endpoint of the line segment.

[0076] Figure 9 shows an example of a polygon that requires two or more virtual points. In Figure 9, observation points are drawn as black circles, and virtual points are drawn as white circles. In the example shown in (a), virtual points vp1a and vp2a are placed in the captured image 20. In the example shown in (b), virtual points vp1b and vp2b are placed in the captured image 20. In the example shown in (c), virtual points vp1c, vp2c, and vp3c are placed in the captured image 20. In the example shown in (d), virtual points vp1d, vp2d, vp3d, vp4d, and vp5d are placed in the captured image 20. In the example shown in (e), virtual points vp1e, vp2e, and vp2f are placed in the captured image 20. The processor 12 sets the range of existence for each virtual point. For example, in the example shown in (a), the processor 12 identifies observation points p1a, p2a, p3a, p4a, p5a, p6a, and p7a from the captured image 20. The processor 12 assigns each of the observation points p1a, p2a, p3a, p4a, p5a, p6a, and p7a to vertex v 0,0 ,v 0,1 ,v 0,2 ,v 2,2 ,v 2,1 ,v 2,0 ,v 1,0 It is then mapped to vertex v. 1,2 The range of existence of the virtual point vp1a corresponding to is set on the line segment p3ap4a. Also, in step S17, the processor 12 sets the range of existence of the vertex v 1,1 The range of existence of the corresponding virtual point vp2a is set to an octagonal region that includes the observation points p1a, p2a, p3a, virtual point vp1a, and observation points p4a, p5a, p6a, p7a as vertices.

[0077] In step S18, the processor 12, acting as the division unit 123, divides the region enclosed by the polygon into N × M quadrilaterals. The processor 12 performs the same processing as in step S15. That is, in step S18, the processor 12 defines N × M quadrilaterals that include as vertices the four observation points or virtual points corresponding to the four vertices contained in each of the N × M rectangles. After step S18, the processor 12 moves the processing to step S19.

[0078] In step S19, the processor 12 determines whether there is a contradiction between the two projected boundaries generated by projectively transforming the boundaries of two adjacent quadrilaterals using two projection transformation matrices corresponding to those two quadrilaterals. The processor 12 performs the same processing as in step S14. For example, in the example shown in Figure 9(a), the processor 12 determines whether there is a contradiction between the two projected boundaries for each of the four boundaries. If the processor 12 determines that there is a contradiction between the two projected boundaries (YES in step S19), it moves the process to step S20. If the processor 12 determines that there is no contradiction between the two projected boundaries for all pairs of adjacent quadrilaterals (NO in step S19), it moves the process to step S22.

[0079] In step S20, the processor 12 determines whether all virtual points coincide with the endpoint of their respective ranges. If the processor 12 determines that all virtual points coincide with the endpoint of their ranges (YES in step S20), it moves the process to step S4. That is, the processor 12 outputs an error and terminates the process. If the processor 12 determines that none of the virtual points coincide with the endpoint of their ranges (NO in step S20), it moves the process to step S21. In the example shown in Figure 7, the processor 12 determines whether virtual point vp1 coincides with the endpoint p25 of its range. In the example shown in Figure 9(a), the processor 12 determines whether virtual points vp1a and vp2a coincide with the endpoints of their respective ranges.

[0080] In step S21, the processor 12 moves one or more virtual points. Refer to Figure 10 to illustrate the movement of virtual points.

[0081] Figure 10 shows an example of the movement of a virtual point. Figure 10 shows a polygon (pentagon) P140 containing observation points p31, p32, p33, p34, and p35 as vertices. Polygon P140 also includes a virtual point vp2 whose existence range is a line segment starting at p35 and ending at p34. Processor 12 moves the coordinates of the virtual point vp2 from the coordinates of observation point p35 along the line segment p35p34 in the direction of observation point p34, according to step S21. Δα is, for example, 1 or a few pixels.

[0082] After step S21, the processor 12 performs the process of step S18 on the polygon containing the one or more moved virtual points. In the example shown in Figure 9(a), the processor 12 fixes the position of virtual point vp1a and moves the position of virtual point vp2a according to step S21. The processor 12 repeatedly moves the position of vp1a when virtual point vp2a coincides with the endpoint and returns the position of virtual point vp2a to the starting point.

[0083] In step S22, the processor 12, acting as the division unit 123, determines the placement of one or more virtual points. That is, the processor 12 determines the placement of one or more virtual points for all pairs of adjacent quadrilaterals, in accordance with its determination that there are no inconsistencies in the boundaries after the two projection transformations. After step S22, the processor 12, acting as the generation unit 124, executes the processes of steps S10 and S11. After that, the processor 12 terminates processing. In the example shown in Figure 7, polygon P130 is quadrilateral Q with observation points p21, p22, p26, and p27 as vertices. 1,1 And quadrilateral Q with vertices p22, p23, p26 and virtual point vp1. 1,2 And quadrilateral Q with vertices p23, p24, p25 and virtual point vp1. 1,3 It is divided into and . As described above, observation points p21, p22, p22, p23, p24, p25, p26, p27 and virtual point vp1 are at vertex v 0,0 ,v 0,1 ,v 0,2 ,v 0,3 ,v 1,3 ,v1,1 ,v 1,2 These points are then mapped to each other. Therefore, the processor 12 maps the quadrilateral Q based on the XY coordinates of these points. 1,1 ,Q 1,2 ,Q 1,3 Rectangle R 1,1 ,R 1,2 ,R 1,3 Three projection transformation matrices are calculated to perform the projection transformation on each of the quadrilateral Q. Then, the processor 12 uses the three calculated projection transformation matrices to transform the quadrilateral Q. 1,1 ,Q 1,2 ,Q 1,3 Each of these is subjected to a projection transformation, and a corrected image R2 containing the three rectangles obtained by the projection transformation is generated.

[0084] <Advantages> As described above, the image correction method according to this disclosure includes step S1 of acquiring a captured image 20 generated by photographing a rectangular answer sheet 40 containing one or more folds. Furthermore, the image correction method includes step S2 of approximating the contour of the target portion in the captured image 20 in which the answer sheet 40 is visible as a polygon. Depending on whether the polygon contains five or more observation points observed on the contour as vertices (YES in step S8), the image correction method further includes a step (including steps S5, S7, S12~S22) of dividing the region enclosed by the polygon into two or more quadrilaterals based on the five or more observation points. Furthermore, the image correction method includes a step (including steps S10, S11) of generating two or more congruent rectangles by projective transformation of the two or more quadrilaterals, and generating a corrected image R2 containing two or more congruent rectangles. As a result, the target portion in which the answer sheet 40 containing one or more folds that divide it into multiple equally spaced rectangular regions is visible can be divided into two or more quadrilaterals with the folds as boundaries. Then, two or more quadrilaterals are projected into two or more congruent rectangles. A corrected image R2 is generated that includes the two or more congruent rectangles obtained by the projective transformation. As a result, in the corrected image R2, not only distortion caused by the shooting angle but also image distortion caused by folds is suppressed.

[0085] The step of dividing the region enclosed by the polygon into two or more quadrilaterals includes step S5, which determines whether each of the five or more observation points corresponds to a corner of the answer sheet 40 or the intersection of a fold and the edge of the answer sheet 40, based on the angle between the two sides adjacent to each of the five or more observation points. Furthermore, the step of dividing the region enclosed by the polygon into two or more quadrilaterals includes step S12, which calculates the number of divisions N in the first direction and the number of divisions M in the second direction for the region, based on the number and arrangement of observation points corresponding to intersections among the five or more observation points. Two or more quadrilaterals include N × M quadrilaterals arranged in the first direction and M in the second direction. Two or more congruent rectangles include N × M rectangles arranged in a grid pattern with N horizontally and M vertically. The step of generating the corrected image R2 includes step S11, which projectively transforms the j-th rectangle in the first direction and the k-th rectangle in the second direction from among the N×M rectangles to the j-th rectangle in the horizontal direction and the k-th rectangle in the vertical direction from among the N×M rectangles. This makes it easy to determine the number of divisions M and N depending on the arrangement of 5 or more observation points.

[0086] The step of dividing the region enclosed by the polygon into two or more quadrilaterals further includes step S13, which involves associating each of the five or more observation points with one of the vertices of N × M rectangles. The step of dividing the region enclosed by the polygon into two or more quadrilaterals further includes step S17, which involves placing one or more virtual points in the captured image 20, each corresponding to one or more vertices of the N × M rectangles that do not correspond to any of the five or more observation points. The j-th quadrilateral in the first direction and the k-th quadrilateral in the second direction has four vertices, each corresponding to one of the four vertices of the j-th rectangle horizontally and the k-th rectangle vertically, from among the five or more observation points and one or more virtual points. The step of dividing the region enclosed by the polygon into two or more quadrilaterals further includes step S19 described above. Step S19 includes calculating two projection transformation matrices for each pair of adjacent quadrilaterals from the N×M quadrilaterals, projecting them onto two rectangles corresponding to the two quadrilaterals from the N×M rectangles, and determining whether there is a contradiction between the two projection transformation boundaries generated by projecting the boundaries of the two quadrilaterals using the two projection transformation matrices. The step of dividing the region enclosed by the polygons into two or more quadrilaterals includes step S21, which moves one or more virtual points depending on whether a contradiction is determined between the two projection transformation boundaries, and step S19, which is repeated after step S21. The step of dividing the region enclosed by the polygons into two or more quadrilaterals also includes step S22, which determines the placement of one or more virtual points depending on whether a contradiction is determined between the two projection transformation boundaries. This ensures that the placement of virtual points corresponding to folds is accurately determined even when observation points corresponding to folds are not observed. As a result, distortion caused by folds is suppressed in the corrected image R2.

[0087] <Variation> The terminal 30 may include an acquisition unit 121, an approximation unit 122, a division unit 123, and a generation unit 124. In this case, the acquisition unit 121 of the terminal 30 acquires the captured image 20 taken by the terminal 10. The approximation unit 122, division unit 123, and generation unit 124 of the terminal 30 then perform the above processing on the captured image 20 acquired from the terminal 10. In this case, the terminal 30 is an example of the "image correction device" of this disclosure.

[0088] The learner may input a capture operation with the answer sheet 40 rotated 90 degrees. In this case, the answer sheet 40 will be captured in the captured image 20 such that the upper left corner of the answer sheet 40 faces the lower left or upper right corner of the captured image 20. Taking this situation into consideration, the processor 12, acting as a division unit 123, uses known text recognition techniques to determine the orientation of the characters captured in the captured image 20. Then, the processor 12 changes the classification method for the first, second, third, and fourth angles of the four observation points assigned the first attribute, according to the determined orientation of the characters. Specifically, if the character is upright, the processor 12 classifies the observation point closest to the origin among the four observation points assigned the first attribute as the first angle. If the character is rotated 90 degrees to the left from the upright position, the processor 12 classifies the observation point closest to the lower left corner of the captured image 20 among the four observation points assigned the first attribute as the first angle. If the orientation of the characters is rotated 90 degrees to the right from the upright position, the processor 12 classifies the observation point closest to the upper right corner of the captured image 20 as the first corner among the four observation points assigned the first attribute. Then, the processor 12 can sequentially classify the three observation points assigned the first attribute that appear when tracing the sides of the polygon counterclockwise from the observation point of the first corner as the second, third, and fourth corners, respectively. As a result, even if the answer sheet 40 is photographed rotated 90 degrees, the processor 12 can generate a corrected image R2 in which the answer sheet 40 is shown upright.

[0089] Furthermore, if the orientation of the characters is rotated 90 degrees to the left from the upright position, the first direction parallel to the line segment whose endpoints are the observation points classified as the second and third angles is roughly parallel to the vertical direction of the captured image 20. The second direction parallel to the line segment whose endpoints are the observation points classified as the first and second angles is roughly parallel to the horizontal direction of the captured image 20. Similarly, if the orientation of the characters is rotated 90 degrees to the right from the upright position, the first direction parallel to the line segment whose endpoints are the observation points classified as the second and third angles is roughly parallel to the vertical direction of the captured image 20. The second direction parallel to the line segment whose endpoints are the observation points classified as the first and second angles is roughly parallel to the horizontal direction of the captured image 20.

[0090] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of this disclosure is indicated by the claims rather than the foregoing description, and all modifications are intended to be equivalent in meaning and scope to the claims. [Explanation of Symbols]

[0091] 10 Terminal, 11 Camera, 12 Processor, 13 Memory, 14 Storage, 15 Communication Interface, 16 User Interface, 20 Captured Image, 30 Terminal, 40 Answer Sheet, 121 Acquisition Unit, 122 Approximation Unit, 123 Segmentation Unit, 124 Generation Unit, 141 Program, 142 Image Folder, R1 Target Area, R2 Corrected Image, R200 Rectangular Frame.

Claims

1. A computer-based image correction method, The process involves obtaining a captured image generated by photographing a rectangular paper medium containing one or more folds, The image correction method further comprises approximating the contour of the portion of the image in which the paper medium is captured as a polygon, wherein the polygon includes five or more observation points observed on the contour as vertices, and the image correction method further comprises Based on the five or more observation points, the region enclosed by the polygon is divided into two or more quadrilaterals, An image correction method comprising generating two or more congruent rectangles by projectively transforming the two or more quadrilaterals, and generating a corrected image that includes the two or more congruent rectangles.

2. The division described above means Based on the angle between the two sides adjacent to each of the five or more observation points, it is determined whether each of the five or more observation points corresponds to a corner of the paper medium or the intersection of the fold and the edge of the paper medium. This includes calculating the number of divisions in the first direction and the number of divisions in the second direction for the region, based on the number and arrangement of observation points corresponding to the intersection among the five or more observation points, When the number of divisions in the first direction is N and the number of divisions in the second direction is M, the two or more quadrilaterals include N × M quadrilaterals arranged in the first direction and M in the second direction, and the two or more congruent rectangles include N × M rectangles arranged in a grid in the horizontal direction and M in the vertical direction. The image correction method according to claim 1, wherein the generation includes projecting the j-th rectangle in the first direction and the k-th rectangle in the second direction from among the N × M rectangles to the j-th rectangle in the horizontal direction and the k-th rectangle in the vertical direction from among the N × M rectangles.

3. The division described above means Each of the five or more observation points is associated with one of the vertices of the N × M rectangles, The division includes placing one or more virtual points in the captured image, corresponding to one or more vertices among the N × M rectangles that do not correspond to any of the five or more observation points, wherein the j-th rectangle in the first direction and the k-th rectangle in the second direction have four points as vertices, corresponding to the four vertices of the j-th rectangle in the horizontal direction and the k-th rectangle in the vertical direction, among the five or more observation points and the one or more virtual points, and the division is as follows: For each pair of adjacent quadrilaterals among the N × M quadrilaterals, calculate two projection transformation matrices that project onto the two rectangles corresponding to the two quadrilaterals among the N × M rectangles. The process involves determining whether there is a contradiction between the two projectively transformed boundary lines generated by projectively transforming the boundary lines of the two quadrilaterals using the two projection transformation matrices, and If it is determined that there is a contradiction in the two projection transformation boundaries, the process of moving one or more virtual points, calculating the two projection transformation matrices, and determining whether or not there is a contradiction is repeated, The image correction method according to claim 2, further comprising determining the placement of one or more virtual points in response to the determination that there is a contradiction between the two projectively transformed boundary lines.

4. An image correction device, An acquisition unit that acquires a captured image generated by photographing a rectangular paper medium containing one or more folds, The image correction device further comprises: an approximation section that approximates the contour of the target portion in which the paper medium is captured in the captured image as a polygon, wherein the polygon includes five or more observation points observed on the contour as vertices, and the image correction device further comprises: Based on the five or more observation points, a division section divides the region enclosed by the polygon into two or more quadrilaterals, An image correction device comprising: a generation unit that generates two or more congruent rectangles by projectively transforming two or more quadrilaterals, and generates a corrected image that includes the two or more congruent rectangles.

5. A program for causing a computer to execute the image correction method described in any one of claims 1 to 3.

Images