Image processing device, image processing method, and program

Abstract

To provide an image processing technology allowing an image suitable for input to image analysis processing to be acquired.SOLUTION: An image processing device identifies at least two lines corresponding to a vertical line on a real space on a first image being a captured image acquired by using a wide-angle lens. The device determines one point on the basis of an intersection of the at least two lines identified and defines a sectorial area including the first image on the basis of the determined one point. The device performs coordinate conversion from the first image to a second image on the basis of the sectorial area, thereby acquiring an image suitable for input to image analysis processing.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing technique for processing captured images. [Background technology]

[0002] There are known techniques for correcting distortion of an image captured by a wide-angle lens. Patent Document 1 discloses a technique for detecting a rectangular area of ​​a subject from a captured image, calculating a distortion rate, and deriving a conversion parameter for correcting the wide-angle lens distortion based on the distortion rate. Patent Document 2 discloses a technique for correcting distortion by switching a correction coefficient according to the position on the captured image in order to reduce lens distortion while maintaining a wide range of display when an image captured by a camera mounted on a vehicle is displayed on a monitor inside the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-165629 A [Patent Document 2] JP 2013-110712 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technologies disclosed in Patent Documents 1 and 2 aim to convert geometric characteristics into ones that look least strange to the human eye, and therefore do not necessarily produce images that are suitable for input into image analysis processing using captured images, for example.

[0005] Therefore, an object of the present invention is to obtain an image suitable for input to image analysis processing. [Means for solving the problem]

[0006] The image processing device of the present invention is InSpecify at least two lines that correspond to vertical lines in real space. Accepts instructions from the user The image processing device includes a first operation means, a correction means for executing a correction process for correcting the first image to a second image based on the positions of the at least two straight lines specified by the first operation means, an evaluation means for evaluating whether or not the at least two straight lines specified by the first operation means are effective in the correction process, and a second operation means for correcting the at least two straight lines specified by the first operation means when the evaluation means evaluates that the at least two straight lines are not effective in the correction process, and the correction means executes the correction process based on the at least two straight lines when the evaluation means evaluates that the at least two straight lines are effective in the correction process. In the correction process, the correction means determines one point based on an intersection point of the at least two straight lines specified by the first operation means, determines a sector area including the first image based on the determined one point, and corrects the first image to the second image based on the sector area. The present invention is characterized by the above. Effect of the Invention

[0007] According to the present invention, an image suitable for input to image analysis processing can be obtained. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration including an image processing apparatus. [Diagram 2] 1A and 1B are diagrams used to explain the relationship between image analysis processing for detecting a person and image distortion. [Diagram 3] 4 is a flowchart of image processing according to the embodiment. [Figure 4] 1 is a diagram used to explain a planar region including images before and after transformation. [Diagram 5] 13 is a flowchart of a process related to a reference line designation by a user. [Figure 6] FIG. 13 is a diagram used to explain evaluation of a reference line. [Figure 7] FIG. 13 is a diagram showing an example of an operation screen prompting a user to check the reference line. [Figure 8] FIG. 13 is a diagram showing an example of an operation screen prompting addition of a reference line. [Figure 9]FIG. 13 is a diagram showing an example of an operation screen indicating that the reference line specification can be completed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the following embodiment does not limit the present invention, and not all of the combinations of features described in the present embodiment are essential to the present invention. Also, the same components will be described with the same reference numerals.

[0010] <First embodiment> FIG. 1 is a diagram showing an example of a schematic configuration of a system including an image processing apparatus according to this embodiment. The image processing device 100 includes a CPU 101, a ROM 102, a RAM 103, a HDD 104, a network control unit 105, a display control unit 106, and an operation control unit 107. These components are connected to each other via a system bus 108, and are capable of transmitting and receiving data. The image processing device 100 is connected to a network 300 via the network control unit 105, and is connected to an imaging device 200 and a storage device 400 via the network 300. The image processing device 100 is also connected to a display device 130 via the display control unit 106. The image processing device 100 is further connected to an operation device 150 via the operation control unit 107.

[0011] The CPU 101 comprehensively controls the operation of the image processing device 100, and controls each component (102 to 107) via a system bus 108. The CPU 101 also executes image processing and image analysis processing according to this embodiment, which will be described later, based on a program. The ROM 102 is a non-volatile memory that stores control programs and the like required for the CPU 101 to execute processing. The HDD 104 is a large-capacity storage device. The HDD 104 stores an OS (operating system), programs for the CPU 101 to execute image processing and image analysis processing according to this embodiment, and various data such as data used in the image analysis processing and analysis result data. The programs may be stored in the HDD 104, as well as in an external memory or a removable storage medium (not shown). The RAM 103 functions as the main memory, work area, and the like of the CPU 101. That is, the CPU 101 loads necessary programs and the like from the ROM 102 or the HDD 104 to the RAM 103 when executing processing, and realizes various functional operations by executing the programs and the like.

[0012] The network control unit 105 is an interface for communicating with external devices via a network 300. The external devices include an imaging device 200 such as a network camera, a storage device 400 for storing recorded data and other data, and other devices connected to the network. The display control unit 106 is an interface for displaying the display data output from the image processing device 100 on the display device 130 . The operation control unit 107 is an interface for transmitting operation instruction information input by a user through the operation device 150 to the image processing device 100. The operation device 150 is, for example, a keyboard, a mouse, or a touch panel. When the operation device 150 is a touch panel, the touch panel may be formed integrally with the screen of the display device 130.

[0013] In the above-described configuration, when the image processing device 100 is powered on, the CPU 101 executes a boot program stored in the ROM 102, and loads the OS from the HDD 104 into the RAM 103 and executes it. Then, under the control of the OS loaded into the RAM 103, the CPU 101 loads an image processing application program according to this embodiment from the HDD 104 into the RAM 103 and executes it. The CPU 101 also loads an image analysis processing application program from the HDD 104 into the RAM 103 and executes it. In this way, the image processing device 100 of this embodiment also functions as an image analysis device.

[0014] Fig. 2(a) is a diagram for explaining the relationship between the distortion of a photographed image and a subject appearing in the image when image analysis processing is performed on the photographed image acquired using a wide-angle lens to detect people, estimate the number of people, etc. Fig. 2(a) shows an example of the result of performing image analysis processing to detect a person as a subject on an image 10 photographed with a wide-angle lens without performing the image processing according to this embodiment.

[0015] In the example of FIG. 2(a), the person 12 surrounded by the dashed line 14 and the person 13 surrounded by the dashed line 15 are shown as people detected as a result of image analysis. In the example of the image 10, three people 11, 12, and 13 are actually captured, but only two people, the person 12 and the person 13, were detected by the image analysis process, and the person 11 was not detected. The people 12 and the person 13 are captured standing upright in the vertical direction in the captured image, so they are detected as people by the image analysis process, whereas the person 11 is captured at an angle, so he is not detected as a person by the image analysis process. In other words, in the case of the image 10 of FIG. 2(a), the person 11 at an angle cannot be detected, so the analysis accuracy of the image analysis process is low. Furthermore, if people are not detected in this way, the analysis accuracy of the image analysis process for estimating the number of people will also be low.

[0016] FIG. 2(b) shows the result of performing the same image analysis process as described above on an image 20 after image processing in which only the oblique person 11 in the image 10 illustrated in FIG. 2(a) is rotated to become the person 21. Since the person 21 after rotation is in a vertically upright state in the image 20, it can be detected as a person by the image analysis process. For this reason, in the example of FIG. 2(b), the person 21 is depicted as a person surrounded by a dashed line 22, which indicates that it has been detected as a result of image analysis. In this way, when a person is obliquely photographed in an image taken with a wide-angle lens, the angle of the person in the image is changed so that the person is in a vertically upright state, so that the person can be detected as a person in the image analysis process. However, in the case of image processing in which only the oblique person in the image 10 is rotated to generate the image 20, the process for determining the region to be rotated is very complicated, and it is not practical to perform this as a preprocessing step for image analysis.

[0017] In this embodiment, the image processing includes a specification process for specifying a line corresponding to a vertical line in real space on a first image captured with a wide-angle lens, and a correction process for correcting the first image to a second image based on the positions of at least two lines specified on the first image. In the correction process, a conversion coefficient for simply geometrically converting the first image so that a subject that is vertical in real space and oblique in the first image becomes vertical is generated based on the positions of at least two lines acquired in the specification process. Then, a geometric conversion process using the conversion coefficient is performed on the first image, correcting the subject such as a person to approach the vertical direction (vertical direction) of the image, and acquiring a second image. This makes it possible to prevent the accuracy of image analysis from being reduced due to the inclination of the subject in the image. That is, for example, even if a person or the like is obliquely photographed on an image captured with a wide-angle lens, it is possible to prevent the accuracy of analysis for detecting the person or the like and estimating the number of the person or the like from being reduced.

[0018] The image 30 in FIG. 2(c) is an image (second image) after the image processing (correction processing by simple geometric transformation) of this embodiment is performed on the image 10 (first image) in FIG. 2(a), and shows the result of image analysis processing to detect a person from the corrected image. In FIG. 2(c), the person 32 corresponds to the person 12 in FIG. 2(a), and similarly, the person 33 corresponds to the person 13, and the person 31 corresponds to the person 11. In the case of FIG. 2(c), among the people 31 to 33 in the image 30, the person 31 in particular is somewhat deformed compared to the person 13 in the original image 10 shown in FIG. 2(a), but is still standing upright in the vertical direction in the image 30. Therefore, in the image 30 in FIG. 2(c), not only the people 32 and 33 but also the person 31 are detected as people by the image analysis processing. That is, in the case of FIG. 2(c), persons 32 and 33 are depicted surrounded by dashed lines 34 and 35, respectively, and person 31 is also surrounded by a dashed line 36, indicating that the person has been detected as a person by image analysis.

[0019] The image processing (geometric transformation processing) that the image processing device 100 of this embodiment performs as preprocessing of the image analysis processing will be described below. 3(a) to 3(c) are flowcharts showing the flow of image processing (geometric transformation processing) executed by the image processing device 100 of this embodiment on a first image acquired from an imaging device 200 such as a network camera or a storage device 400 via a network 300. Note that each process in the following flowcharts is realized by the CPU 101 of the image processing device 100 executing a program according to this embodiment.

[0020] In step S301, the CPU 101 accepts input of an image to be processed. The image to be processed is assumed to be an image (first image) captured using a wide-angle lens. Next, in step S302, CPU 101 specifies a reference line, which is a straight line that is presumed to indicate the vertical direction of the subject in real space, on the input image accepted in step S301 and is used in the subsequent calculation of conversion coefficients.

[0021] FIG. 3B is a flowchart showing details of the reference line determination process in step S302. In step S308 in FIG. 3B, the CPU 101 detects straight lines in the image. The method of detecting straight lines may be any of various generally well-known methods. As an example, the CPU 101 converts the input image into a grayscale image and extracts edge pixels that are components of edges by using differential information of each pixel value. Next, the CPU 101 virtually defines a plurality of straight lines at various positions and angles, and performs a Hough transform to select straight lines on which many of the extracted edge pixels are present. At this time, even if the subject is a straight line, it may be distorted in the captured image due to distortion caused by the lens, so a tolerance for determining that edge pixels are present on the virtually defined straight line is set according to the shooting conditions. At this time, it is desirable to remove straight lines that would not have been necessary to be selected from the selected straight lines as noise. For example, the CPU 101 cancels the selection of straight lines on which edge pixels are distributed only in a very narrow range, that is, very short straight lines.

[0022] Next, in step S309, CPU 101 extracts lines that satisfy a predetermined condition that can be assumed to be vertical lines in real space from the group of lines detected in step S308. The predetermined condition that can be assumed to be a vertical line is defined on the premise that the input image is an image captured with a wide-angle lens. That is, a line drawn at an angle from the upper right to the lower left on the right side of the center of the image, and a line drawn at an angle from the upper left to the lower right on the left side of the center of the image are extracted as the vertical line of the subject. Note that the accuracy can be further improved by defining the angle range for determining a vertical line according to the position in the image, taking into account the shooting conditions.

[0023] Next, in step S310, the CPU 101 identifies a line to be used to calculate coefficients of a geometric transformation to be performed later from among the lines estimated to be vertical lines in the real space in step S309, and determines it as a reference line. At this time, if the number of lines estimated to be vertical lines in the real space is less than a predetermined number (for example, two), the CPU 101 sets all of these lines as reference lines. On the other hand, if the number of lines estimated to be vertical lines in the real space is equal to or greater than the predetermined number, the CPU 101 selects a representative line among multiple lines having similar angles and positions relative to the screen and sets it as the reference line. Note that, as examples of lines having similar angles and positions relative to the screen, lines within a threshold angle range or a threshold position range may be considered as similar lines. The CPU 101 also removes peculiar lines that can be determined as noise from the candidates for the reference line. For example, if the direction (angle) of a line is clearly different from many neighboring lines, the CPU 101 determines that the line is a peculiar line and removes it from the candidates for the reference line. Whether or not a line is clearly different from neighboring lines may be determined based on, for example, a threshold range of angles, etc. In this manner, CPU 101 determines, from the input image (first image), at least two or more reference lines that are lines corresponding to vertical lines in real space and have different angles within the image.

[0024] After the process of step S310 in FIG. 3B, that is, after the process of step S302 in FIG. 3A, CPU 101 advances the process to step S303 in FIG. 3A. When proceeding to step S303, CPU 101 sets a sectorial area including the captured image (first image) as a virtual planar area before geometric transformation based on the two or more reference lines having different angles in the image determined in step S302. CPU 101 then calculates the center coordinates of the sectorial area (coordinates corresponding to the corner of the fan). The setting of the sectorial area based on the two or more reference lines having different angles in the image and the center coordinates of the sectorial area will be described in detail later.

[0025] FIG. 3C is a detailed flowchart of the process in step S303 for setting a sector region based on two or more reference lines having different angles in the image and calculating the coordinates of its center. In step S311 of FIG. 3C, CPU 101 calculates the coordinates of the intersections of lines extended from two or more reference lines having different angles determined in step S302 for all combinations of these reference lines.

[0026] Next, in step S312, CPU 101 extracts the intersections whose coordinates are within a predetermined range from among the intersections calculated in step S311. If only one intersection is calculated in step S311, CPU 101 extracts the one intersection. On the other hand, if there are multiple intersections calculated in step S311, CPU 101 extracts the intersections that are within a predetermined range from among the multiple intersections. Here, the predetermined range is a range of a predetermined width centered on the center of either the horizontal or vertical width of the image. For example, if the horizontal width direction of the image is the x-axis direction and the vertical width direction of the image is the y-axis direction (axis perpendicular to the x-axis direction), the predetermined range is a range sandwiched between two straight lines parallel to the y-axis, each of which has a predetermined distance on the x-axis, sandwiching a straight line in the y-axis direction that passes through the center of the horizontal width of the image. In this case, CPU 101 extracts the intersections whose x-coordinates are within the predetermined range. In this example, the predetermined range is within a predetermined distance from a vertical line in the y-axis direction that passes through the center of the width of the image, but the predetermined range may be within a predetermined distance from a vertical line in the x-axis direction that passes through the center of the height of the image. CPU 101 may switch between the predetermined range within a predetermined distance from a vertical line that passes through the center of the width of the image and the predetermined range within a predetermined distance from a vertical line that passes through the center of the height of the image, as appropriate, depending on the shooting conditions, etc.

[0027] Next, in step S313, CPU 101 determines whether or not an intersection was extracted in step S312 and branches the process. If no intersection was extracted, CPU 101 proceeds to step S316, where it generates information indicating that the central coordinates of the sectorial region have not been determined. On the other hand, if one or more intersections have been extracted, CPU 101 proceeds to step S314.

[0028] When the process proceeds to step S314, CPU 101 determines a representative value of the y coordinate based on the extracted intersection coordinates. CPU 101 determines, for example, the average value or median value of the y coordinates of all the extracted intersections as the representative value. The method of determining the representative value of the y coordinate is not limited, and CPU 101 may, for example, determine a y coordinate in a vicinity where many intersections are concentrated based on the distribution of the y coordinates of all the extracted intersections and use this as the representative value.

[0029] Then, in the next step S315, CPU 101 determines the central coordinate of the sector area. In this embodiment, CPU 101 sets the representative value of the y coordinates determined in step S314 as the y coordinate of the central coordinate of the sector area. CPU 101 also sets the x coordinate corresponding to the center of the image in the width direction as the x coordinate of the central coordinate of the sector area. CPU 101 then generates information indicating that the central coordinate of the sector area has been determined. In this manner, the central coordinate calculation process of the sector area is performed.

[0030] After the process of the flowchart in FIG. 3C, that is, after step S303 in FIG. 3A, CPU 101 advances the process to step S304 in FIG. 3A. When the process proceeds to step S304, CPU 101 branches the process depending on whether the center coordinates of the sector area have been determined. That is, if the center coordinates of the sector area have been determined, CPU 101 proceeds to step S305, and if the center coordinates of the sector area have not been determined, CPU 101 proceeds to step S307.

[0031] When the process proceeds to step S305, CPU 101 calculates a conversion coefficient used in the geometric conversion for correcting the first image to the second image. At this time, CPU 101 defines a planar area before and after the geometric conversion, and generates information for mapping coordinates. CPU 101 determines the size of the second image based on the sector area and the first image, and performs mapping processing for converting the coordinates of the sector area to the second image so as to match the determined size. Note that the calculation of the conversion coefficient may be performed when the central coordinates of the sector are determined. After that, in step S306, CPU 101 performs coordinate conversion by mapping processing using the conversion coefficients calculated in step S305, thereby performing geometric conversion of the first image to correct it to a second image.

[0032] Figures 4(a) to 4(d) are diagrams used to conceptually explain the processing from calculating the center coordinates of the sector to geometric transformation, and the processing until a second image is obtained in which diagonal lines in a first image are corrected to vertical lines by the geometric transformation. 4(a) is a diagram showing the relationship between an input image, which is a first image, and a sector, and the input image is shown in the diagram as an original image area 40. The sector area 42 is a sector-shaped area that circumscribes the original image area 40, and the position corresponding to the pinnacle of the sector is the central coordinates 41 of the sector. That is, in the above-mentioned step 303, the central coordinates 41 in the diagram are calculated as the central coordinates of the sector.

[0033] FIG. 4(b) is a diagram showing a planar region after the planar region consisting of the original image region and the fan-shaped region is geometrically transformed. In this embodiment, a geometric transformation is performed to transform the fan-shaped region 42 in FIG. 4(a) into a rectangular region. The transformed rectangular region 44 is a rectangular region after the fan-shaped region 42 is geometrically transformed, and the transformed image region 43 is a region after the original image region 40 is geometrically transformed. The size and shape (aspect ratio) of the transformed rectangular region 44 may be the same as those of the original image region 40, or may be adjusted according to the shooting conditions and the resolution required for the subsequent analysis process. For example, the larger the central angle of the fan-shaped region 42, the greater the proportion of the size of the arc part of the fan shape that is relatively reduced by the geometric transformation. Therefore, in order to ensure the resolution required for the analysis process, the size of the transformed rectangular region 44 is adjusted to be larger than that of the original image region 40.

[0034] Mapping of the sector region 42 onto a rectangular region such as the transformed rectangular region 44 is performed by determining coordinates on the sector region 42 that correspond to the coordinates defined at equal intervals on the transformed rectangular region 44. The coordinates on the sector region 42 may be determined at equal intervals with respect to the central angle and the radius of the arc with respect to the central coordinate 41 as the reference, but depending on the shooting conditions, the radial intervals may be changed according to the depth of the subject. By performing such a geometric transformation, lines 45 and 46 which correspond to vertical lines in real space within the sector region 42 shown in FIG. 4(c) are corrected to lines 47 and 48 which are approximately vertical on the image of the transformed rectangular region 44 as shown in FIG. 4(d).

[0035] Returning to the flowchart of FIG. If it is determined in step S304 that the central coordinates of the sector have not been determined and the process proceeds to step S307, CPU 101 acquires information on the additional reference line input by the user via a user interface (UI). In the present embodiment, the additional reference line is given by input from the user, but the method is not important. For example, the user may directly input coordinate information of the start point and the end point on the original image, and CPU 101 may acquire information on the line specified by the coordinate information of the start point and the end point as information representing the additional reference line. Also, for example, the original image may be displayed on display device 130, and the user may specify a corresponding location on the screen via operation device 150, and CPU 101 may convert the specified information into coordinate information of the additional reference line. If the process of step S307 acquires enough reference lines to determine the central coordinates of the sector, CPU 101 may proceed to the process of step S305 and subsequent steps.

[0036] In the image processing device 100 of the first embodiment, two or more reference lines that are assumed to be vertical in real space are defined on a first image captured with a wide-angle lens, and a correction process is performed to geometrically transform the first image based on the reference lines to obtain a second image. In this way, the image processing device 100 of the first embodiment can obtain an image suitable for input to image analysis processing.

[0037] Second Embodiment In the first embodiment, an example is described in which a distorted image captured with a wide-angle lens is geometrically transformed to have characteristics suitable for analysis processing. In the second embodiment, an operation screen and an operation screen display process are described that allow a user to directly specify a reference line for determining a conversion coefficient (correction coefficient) used in the geometric transformation.

[0038] FIG. 5 is a flowchart of a process related to an operation screen for a user to specify a reference line. In step S501, CPU 101 monitors whether the user has performed an operation to add a reference line, based on information input from operation device 150 via operation control unit 107. If the user has performed an operation to add a reference line, CPU 101 advances the process to step S502, where it evaluates the reference line added by the user. In the following description, the reference line added by the user will be referred to as an added reference line.

[0039] FIG. 6 is a diagram used to explain the evaluation process of the additional reference line in step S502. In Fig. 6, reference lines 610, 620, and 630 shown in thick lines respectively represent additional reference lines specified by the user. Also, dashed lines in Fig. 6 indicate extensions of the added reference lines 610, 620, and 630 respectively. Furthermore, Fig. 6 illustrates intersections 611, 612, 621, 622, 631, and 632 where the extensions of the additional reference lines intersect with each side of the image 600. For example, the intersections of the extension of the additional reference line 610 with each side of the image 600 are intersections 611 and 612.

[0040] Here, since the reference lines in this embodiment correspond to the vertical direction in real space, one of two intersections between the extensions of each reference line and each side of the captured image should normally intersect with a specified side of the captured image. For example, if the lateral direction of image 600 in Fig. 6 corresponds to the horizontal direction in real space, one of the intersections between the extensions of each reference line and each side of the image should intersect with the bottom side of image 600.

[0041] In the example of FIG. 6, the extension of the additional reference line 610 intersects with the bottom side of the image 600 at intersection 612, and the extension of the additional reference line 620 intersects with the bottom side of the image 600 at intersection 622. Therefore, these additional reference lines 610 and 620 can be determined to be lines representing the vertical direction of the subject in real space. In this embodiment, of the reference lines added by the user, a reference line that can be determined to be a line representing the vertical direction of the subject in real space is called a normal reference line. On the other hand, the extension of the additional reference line 630 intersects with the sides of the image 600 at intersections 631 and 632, but neither of the intersections 631 and 632 is on the bottom side of the image 600. In other words, since the extension of the additional reference line 630 does not intersect with the bottom side of the image 600, it can be determined that the additional reference line 630 is not a normal reference line corresponding to the vertical direction of the subject in real space, but is an inappropriate line as a reference line (i.e., an abnormal reference line).

[0042] In this manner, in step S502, CPU 101 evaluates whether the additional reference line is a normal reference line by determining whether the additional reference line intersects with a predetermined side of the image (the bottom side in the example of FIG. 6). That is, in the example of FIG. 6, CPU 101 evaluates that, of additional reference lines 610, 620, and 630, additional reference lines 610 and 620 are normal reference lines, and evaluates additional reference line 630 as not a normal reference line (an abnormal reference line).

[0043] Then, in step S503, CPU 101 determines whether all additional reference lines are evaluated as normal reference lines in the evaluation in step S502. In other words, CPU 101 determines whether there is any additional reference line evaluated as not normal in the evaluation process of step S502. If CPU 101 evaluates all additional reference lines as normal in step S503, it proceeds to step S507, and on the other hand, if there is any additional reference line evaluated as not normal, it proceeds to step S504. In the example of FIG. 6, since additional reference line 630 is evaluated as not normal, it proceeds to the process of step S504.

[0044] In step S504, the CPU 101 controls the display control unit 106 to display a message prompting for confirmation on the operation screen of the display device 130. FIG. 7 is a diagram showing an example of an operation screen on which a confirmation message is displayed. As shown in FIG. 7, an image display area 710, an OK button 730, a cancel button 740, and a message 720 are arranged on a screen 700 of the display device 130. The captured image is displayed in the image display area 710, and the additional reference lines 610, 620, and 630 designated by the user are displayed on the image. The message 720 is a text message that prompts the user to check the additional reference line that is evaluated as not being a normal reference line. In addition, at this time, the additional reference line that is the subject of the user's check is displayed differently from the other normal reference lines in order to clearly indicate that it is a reference line that the user needs to check. In the case of FIG. 7, the additional reference lines 610 and 620 that are evaluated as normal reference lines are displayed, for example, as solid lines, while the additional reference line 630 that is evaluated as not being a normal reference line is displayed, for example, as a dotted line. Note that the display method of the additional reference line that is the subject of user confirmation is not limited to a dotted line, and any method may be used as long as it can be distinguished from the normal reference line, such as changing the display color or blinking. By displaying an operation screen such as that shown in FIG. 7, the user can perform an operation such as correction or change on the additional reference line being confirmed, or can confirm it but not make any corrections.

[0045] In the next step S505, CPU 101 monitors information input from operation device 150 via operation control unit 107, and waits until a user operation is performed on the operation screen for checking the reference line as shown in Fig. 7. If there is no user operation, CPU 101 returns the process to step S504, and if there is an operation, CPU 101 proceeds to step S506.

[0046] When the process proceeds to step S506, CPU 101 determines whether the user has performed an operation to correct the reference line from operation device 150 via operation control unit 107, and branches the process accordingly. If an operation to correct the reference line has been performed, CPU 101 returns the process to step S502 and performs evaluation processing on the reference line after it has been corrected by the user. On the other hand, if the user has not performed an operation to correct the reference line and has performed a user operation indicating that the reference line should be confirmed and the process should proceed, CPU 101 proceeds to step S507.

[0047] In step S507, CPU 101 determines whether or not two or more normal reference lines have been set, and if so, proceeds to step S508. On the other hand, if there are fewer than two normal reference lines, additional reference lines are required, so CPU 101 proceeds to step S510.

[0048] In step S508, the CPU 101 calculates conversion coefficients using two or more normal reference lines. The processing here is the processing from step S303 to step S305 described in the first embodiment, and the description thereof will be omitted. Next, in step S509, CPU 101 determines whether or not the calculation of the conversion coefficients has been completed normally, and branches the process accordingly. If the calculation of the conversion coefficients has been completed normally, CPU 101 advances the process to step S511. On the other hand, if the calculation of the conversion coefficients has not been completed normally, CPU 101 advances the process to step S510 since it is necessary to reconsider the setting of the reference line.

[0049] In step S510, the CPU 101 controls the display control unit 106 to display an operation screen that prompts the user to add a reference line. 8(a) to 8(d) are diagrams used to explain an example of an operation screen for specifying a reference line that is displayed on display device 130 by the process in step S510.

[0050] FIG. 8(a) is a diagram showing an example of an operation screen 800 when a reference line is specified, and shows a state in which a reference line 811 corresponding to the vertical direction of a subject in real space has already been set. The reference line 811 is assumed to be an additional reference line specified by the user, but may be a reference line specified based on a subject in an image, as described in the first embodiment. An image display area 810, an OK button 820, a cancel button 830, a message 840, and the like are arranged on the operation screen 800. A captured image is displayed in the image display area 810, and the user can specify a reference line at any position on the image display area 810. In the example of FIG. 8(a), only one reference line is currently specified, and is displayed as a straight reference line 811. Also, as described in the first embodiment, two or more reference lines are required to calculate a conversion coefficient. That is, as in the example of FIG. 8(a), when less than two reference lines are currently specified, a new reference line needs to be additionally specified. For this reason, a message 840 is displayed on the operation screen 800 to prompt the user to additionally specify a reference line.

[0051] At this time, a display is provided in the image display area 810 to clearly indicate to the user the area in which it is recommended to add a new reference line. FIG. 8(a) shows an example in which an additional designation recommended area 812 surrounded by a dashed line is displayed in the image display area 810 as an area in which it is recommended to add a new reference line. The additional designation recommended area 812 is an area that is estimated to be suitable for determining the central coordinates of the above-mentioned sector shape by combining a new reference line with the already-set reference line 811 when a new reference line is added to any position in the area. As in the example of FIG. 8(a), when a new reference line is not added, the conditions for calculating the conversion coefficient are not satisfied, and therefore, although the OK button 820 is displayed, its function is disabled.

[0052] 8(b) to 8(d) are diagrams for explaining an example of a setting method for setting an additional designation recommended region 812 in an image display region 810 based on a set reference line 811. Region 850 in FIG. 8(b), region 860 in FIG. 8(c), and region 870 in FIG. 8(d) each represent the image display region 810 in FIG. 8(a). A straight line 851 in the image display region 850 in FIG. 8(b) represents the set reference line and its extension line. Similarly, a straight line 861 in the image display region 860 in FIG. 8(c), and a straight line 871 in the image display region 870 in FIG. 8(d) each represent the set reference line and its extension line.

[0053] The newly added reference line is used in combination with the already-set reference line when calculating the central coordinates of the sector. For this reason, it is desirable that the recommended additional designation area is an area in which, when a new reference line is added within the area, the most suitable reference line for determining the central coordinates of the sector is set when the newly added reference line is combined with the already-set reference line. In this embodiment, the recommended additional designation area is set as an area in which the intersection of the extension of the newly added reference line and the bottom side of the image is in a position symmetrical with respect to the intersection position of the extension of the already-set reference line and the bottom side of the image, with respect to the center point of the bottom side of the image.

[0054] In addition, if the subject on the image used as a guide for specifying the reference line is separated on the left and right sides of the image, it is considered that it is easy to estimate the characteristics of the entire image, that is, the characteristics that indicate the degree of distortion of the image. Therefore, when specifying a new reference line, it is preferable to specify a reference line having an x ​​coordinate in the direction of the center of the image rather than the intersection position of the extension of the already set reference line and the bottom side of the image. Therefore, for example, in the case of the image display area 850 shown in FIG. 8(b), the additional designation recommended area for a new reference point is set to an area 852 on the side of the center of the image, with the x coordinate of the intersection of the straight line 851 representing the already set reference line and the bottom side of the image as the boundary. Also, for example, in the case of the image display area 860 shown in FIG. 8(c), the additional designation recommended area is set to an area 862 on the side of the center of the image, with the x coordinate of the intersection of the straight line 861 representing the already set reference line and the bottom side of the image as the boundary. For example, in the case of image display area 870 shown in FIG. 8(d), the x-coordinate of the intersection between straight line 871 representing the specified reference line and the bottom side of the image is set as the boundary, and area 872 is set on the central side of the image.

[0055] Returning to the flowchart of FIG. Next, in step S509, if it is determined that the calculation of the conversion coefficients has been completed successfully and the process proceeds to step S511, the CPU 101 controls the display control unit 106 to display on the display device 130 an operation screen indicating that the reference line designation operation can be completed.

[0056] FIG. 9 is a diagram showing an example of a reference line designation screen 900 displayed on the display device 130 by the process of step S511, and shows the screen display state when a reference line required for calculating the conversion coefficients has been correctly designated. 9, an image display area 910, an OK button 930, a cancel button 940, and a message 920 are arranged on a reference line designation screen 900. The captured image is displayed in the image display area 910, and reference lines 610 and 620 that can be used to calculate conversion coefficients are displayed. The message 920 is a text message indicating that a sufficient number of reference lines have been designated and that this process can be completed. The OK button 930 at this time has a function that enables the user to confirm the designation of the reference lines and end the process by pressing it, etc.

[0057] After step S511, the process proceeds to step S512, where CPU 101 monitors the user's operation of editing (adding, correcting) the reference line, based on information input from operation device 150 via operation control unit 107. Here, if CPU 101 detects a user's operation of editing the reference line, it returns the process to step S502, and if an editing operation of the reference line is performed after step S502, it continues the same process as described above for the edited reference line.

[0058] Furthermore, while monitoring the editing of the reference line, in step S513, CPU 101 monitors whether the user has pressed OK button 930 from operation device 150 via operation control unit 107. If OK button 930 has not been pressed, CPU 101 returns the process to step S512, and if OK button 930 has been pressed, ends the reference line designation process of the flowchart in FIG. As described above, according to the second embodiment, it is possible to provide an operation screen that guides the user to correctly specify a reference line.

[0059] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0060] 100: image processing device, 101: CPU, 102: ROM, 103: RAM, 106: display control unit, 130: display device, 200: imaging device, 400: storage device

Claims

1. a first operation means for receiving an instruction from a user to specify at least two straight lines corresponding to vertical lines in a real space in the first image; a correction unit that executes a correction process to correct the first image to a second image based on the positions of the at least two straight lines specified by the first operation unit; an evaluation means for evaluating whether or not the at least two straight lines designated by the first operation means are valid in the correction process; a second operation means for correcting the at least two straight lines designated by the first operation means when the evaluation means evaluates the at least two straight lines as being invalid in the correction process; The correction means is executing the correction process based on the at least two straight lines when the evaluation means evaluates the at least two straight lines as being effective in the correction process; In the correction process, the correction means determining a point based on an intersection of the at least two straight lines designated to the first operating means; determining a sector region including the first image based on the determined point; 23. An image processing apparatus comprising: a processor for correcting the first image to the second image based on the sector area.

2. The correction means, in the correction process, defining the sector area circumscribing the first image, the sector area having the determined point as a center coordinate of the sector; determining a size of the second image based on the sector area and the first image; 2. The image processing apparatus according to claim 1, wherein coordinate conversion from the sector area to the second image is performed in accordance with the size.

3. The first operation means extracts at least three straight lines that are assumed to correspond to vertical lines in real space from the first image; The correction means is Obtaining a plurality of intersection points of the at least three extracted straight lines; 3. The image processing apparatus according to claim 1, wherein a representative value of a plurality of intersections having coordinates within a predetermined range among the plurality of acquired intersections is determined as the single point.

4. The image processing device according to claim 3, wherein the correction means sets the representative value to an average or median value of the coordinates of a plurality of intersections whose coordinates are within a predetermined range, or a coordinate at which the distribution of the coordinates of the plurality of intersections whose coordinates are within the predetermined range is concentrated.

5. 5. The image processing device according to claim 3, wherein the predetermined range is a range of a predetermined width centered on a straight line passing through the center of either the horizontal or vertical width of the image.

6. 2. The image processing apparatus according to claim 1, wherein said evaluation means, when evaluating that said at least two straight lines are not effective in said correction processing, displays a message requesting a user to confirm.

7. The image processing device according to claim 6, characterized in that the evaluation means evaluates that the at least two straight lines specified by the user using the first operation means are effective in the correction processing when the at least two straight lines intersect with a specified side of the first image.

8. 8. The image processing device according to claim 6, wherein, when the evaluation means evaluates that the at least two straight lines specified by the user using the first operation means are not effective in the correction process, the evaluation means displays a message requesting the user to add a straight line.

9. The image processing device according to any one of claims 6 to 8, characterized in that when the evaluation means evaluates that the at least two straight lines specified by the user using the first operation means are not effective in the correction process, it displays a display indicating a recommended area in which to add straight lines.

10. The image processing device described in claim 9, characterized in that the evaluation means sets as the recommended area an area in which the intersection points between at least two straight lines newly specified by the user and a specified side of the first image are located symmetrically with respect to the midpoint of the specified side of the first image with respect to the intersection points between at least two straight lines already set and the specified side of the first image.

11. The image processing device according to claim 10, characterized in that the evaluation means sets an area toward the center of the first image as the recommended area, with the intersection of an already set straight line and a specified side of the first image as the boundary.

12. 2. The image processing device according to claim 1, wherein the evaluation means evaluates that the at least two straight lines specified by the user via the first operation means are effective in the correction process when the at least two straight lines intersect with a specified side of the first image.

13. An image processing method executed by an image processing device, comprising: a first operation step of receiving an instruction from a user to specify at least two straight lines corresponding to vertical lines in a real space in the first image; a correction step of executing a correction process for correcting the first image to a second image based on the positions of the at least two straight lines designated in the first operation step; an evaluation step of evaluating whether or not the at least two straight lines designated in the first operation step are valid in the correction process; a second operation step of correcting the at least two straight lines designated in the first operation step when the at least two straight lines are evaluated as not valid in the correction process in the evaluation step, In the correction step, performing the correction process based on the at least two straight lines when the evaluation step evaluates the at least two straight lines as being valid for the correction process; In the correction process, determining a point based on an intersection of the at least two straight lines specified in the first operation step; determining a sector region including the first image based on the determined point; An image processing method comprising: correcting the first image to the second image based on the sector region.

14. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 12.

Images