Distortion correction coefficient calculation method and program

Abstract

To calculate a distortion correction coefficient from a photographic image of a general environment with high accuracy.SOLUTION: A processing part 1a of an information processing device 1 extracts a plurality of segments 21 to 23 by executing segment extraction processing to each of a plurality of photographic images 11, 12 of different photographing times. The processing part 1a calculates an optimum value of a distortion correction coefficient by executing distortion correction processing to the extracted plurality of segments 21 to 23 while changing a setting value of the distortion correction coefficient to evaluate a processing result. The number of segments to be used to calculate the optimum value of the distortion correction coefficient can be increased by extracting the plurality of segments 21 to 23 from the plurality of photographic images 11, 12. As a result, the distortion correction coefficient can be calculated with high accuracy even when an image having a small number of segments is photographed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a distortion correction coefficient calculation method and a program. [Background technology]

[0002] Technologies that estimate the position, speed, and distance of an object from a captured image are becoming widespread. Such estimation requires the camera parameters of the camera used to capture the image. One of the camera parameters (particularly the internal parameters) is the distortion correction coefficient for correcting the lens distortion aberration.

[0003] One method for calculating the distortion correction coefficient is to capture a special image pattern such as a checkerboard and calculate the distortion correction coefficient based on the position of the image pattern on the image. However, this method requires that the distortion correction coefficient be calculated using a special image pattern for each camera, which is inconvenient. In addition, since errors can occur in the calculation results depending on how the image is captured, a certain level of skill is required for the work.

[0004] In response to this, a method has been proposed in which a distortion correction coefficient is calculated using an image captured in a general environment without using a dedicated image pattern. For example, a technique has been proposed in which an optimal distortion correction coefficient is estimated based on how straight line components in an image appear when a distortion correction process is performed on a captured image.

[0005] As a related technique, a method has been proposed in which line segments are extracted from an image, a distortion aberration coefficient is calculated for each extracted line segment by assuming that a portion of the straight line is distorted, and the calculation results are statistically processed to estimate an optimal distortion aberration coefficient. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2004-86747 A [Non-patent literature]

[0007] [Non-Patent Document 1] Yuta Kanuki, Doctoral thesis, Gunma University Graduate School, "Research on automatic correction technology for camera radial distortion using environmental structure," December 2015 Summary of the Invention [Problem to be solved by the invention]

[0008] As described above, in the technology for estimating the distortion correction coefficient based on the appearance of the straight line components in the image when the captured image is subjected to the distortion correction process, there is a problem that the estimation accuracy decreases when the number of straight line components in the captured image is small. For example, an image captured by an on-board camera while driving in an area with few buildings contains few straight line components, and the distortion correction coefficient cannot be estimated with high accuracy from such a captured image.

[0009] According to one aspect, the present invention has an object to provide a distortion correction coefficient calculation method and a program capable of calculating a distortion correction coefficient with high accuracy from an image captured in a general environment. [Means for solving the problem]

[0010] One proposal provides a distortion correction coefficient calculation method in which a computer executes the following process: In this distortion correction coefficient calculation method, the computer executes a line segment extraction process for each of a plurality of captured images captured at different times to extract a plurality of line segments, executes a distortion correction process for the plurality of line segments while changing a set value of the distortion correction coefficient, and evaluates the processing result to calculate an optimal value of the distortion correction coefficient.

[0011] In addition, in one proposal, a distortion correction coefficient calculation program is provided that causes a computer to execute a process similar to the distortion correction coefficient calculation method described above. Effect of the Invention

[0012] On the one hand, the distortion correction coefficients can be calculated with high accuracy from images taken in a typical environment. [Brief description of the drawings]

[0013] [Figure 1] 1 illustrates a configuration example and a processing example of an image processing system according to a first embodiment; [Diagram 2] FIG. 13 illustrates an example of a configuration of an image processing system according to a second embodiment. [Diagram 3] FIG. 13 is a diagram showing an example of a checkerboard. [Figure 4] 1A and 1B are diagrams illustrating examples of images before and after distortion correction. [Diagram 5] FIG. 13 is a diagram showing an example of a captured image having a small number of line segments. [Figure 6] FIG. 13 is a diagram showing an example of line segment extraction from a plurality of captured images. [Figure 7] FIG. 2 is a diagram illustrating an example of the configuration of processing functions included in an image processing apparatus. [Figure 8] 11A and 11B are diagrams illustrating an example of data stored in a line segment data storage unit; [Figure 9] 11 is an example of a flowchart illustrating an entire distortion correction coefficient calculation process. [Figure 10] 13 is a flowchart illustrating an example of a line segment adding process. [Figure 11] 13 is a flowchart illustrating an example of a similar line segment determination process. [Figure 12] 13 is a flowchart illustrating an example of a distortion correction coefficient estimation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. First Embodiment 1 is a diagram showing an example of the configuration and processing of an image processing system according to a first embodiment. The image processing system shown in FIG.

[0015] The information processing device 1 includes a processing unit 1a. The processing unit 1a is, for example, a processor. In this case, the processing of the processing unit 1a, which will be described later, can be realized by the processor executing a predetermined program.

[0016] The information processing device 1 acquires an image captured by the camera 2, and uses the acquired captured image to calculate a distortion correction coefficient for correcting distortion of the lens of the camera 2. The information processing device 1 is capable of calculating the distortion correction coefficient from a captured image obtained by having the camera 2 capture an image of a general environment, without having the camera 2 capture a dedicated image pattern for calculating the distortion correction coefficient. For example, when the camera 2 is mounted on a vehicle traveling on a road, the information processing device 1 can calculate the distortion correction coefficient from an image captured by the camera 2 of the surrounding environment, such as a road, from the traveling vehicle.

[0017] The processing unit 1a of the information processing device 1 calculates the distortion correction coefficient in the following procedure. The processing unit 1a acquires a plurality of captured images taken at different times by the camera 2. For example, the processing unit 1a acquires a plurality of captured images taken continuously in time series. The processing unit 1a extracts a plurality of line segments by executing a line segment extraction process on each of the acquired plurality of captured images. The extraction of the line segments can be performed using a known line detection method such as a Hough transform or an LSD (Line Segment Detector).

[0018] In the example of Fig. 1, captured images 11 and 12 captured at different times are acquired, a line segment 21 is extracted from captured image 11, and line segments 22 and 23 are extracted from captured image 12. Moreover, a composite image 13 shown in Fig. 1 is obtained by combining and displaying the extracted line segments 21 to 23 on the same image. However, in reality, it is not necessary to generate composite image 13, and it is sufficient to at least acquire position information in the image of the extracted line segments 21 to 23.

[0019] The processing unit 1a performs distortion correction processing on the extracted line segments 21 to 23 while changing the set value of the distortion correction coefficient. In the example of Fig. 1, distortion correction processing is performed on the line segments 21 to 23 using each of m set values ​​X1, X2,...,Xm. If lens distortion occurs, the positions and shapes of the line segments 21 to 23 may change after the distortion correction processing is performed.

[0020] The processing unit 1a evaluates the results of such distortion correction processing to calculate the optimal value of the distortion correction coefficient. For example, every time the processing unit 1a executes the distortion correction processing using different setting values, it calculates the entropy of the directional histogram for the line segments 21 to 23 after the execution of the distortion correction processing, and calculates the optimal value of the distortion correction coefficient based on the calculated entropy.

[0021] Here, in the above calculation process for calculating the optimum value of the distortion correction coefficient using the extracted line segments, if the number of extracted line segments is small, the distortion correction coefficient cannot be calculated with high accuracy. For example, only one line segment 21 is extracted from the photographed image 11, and only two line segments 22 and 23 are extracted from the photographed image 12. For this reason, if the optimum value of the distortion correction coefficient is calculated using only the photographed image 11 or only the photographed image 12 in the above procedure, the distortion correction coefficient cannot be calculated accurately.

[0022] In contrast, in the present embodiment, the optimum value of the distortion correction coefficient is calculated using line segments extracted from a plurality of captured images captured at different times. By extracting line segments from a plurality of captured images captured at different times, the number of extracted line segments can be increased compared to the case where a single captured image is used. Furthermore, by calculating the optimum value of the distortion correction coefficient using many line segments, it is possible to improve the calculation accuracy of the optimum value of the distortion correction coefficient compared to the case where a single captured image is used. Therefore, it becomes possible to calculate the distortion correction coefficient with high accuracy from images captured in a general environment.

[0023] The above-mentioned line segment extraction process and distortion correction coefficient calculation process by the processing unit 1a may be executed, for example, inside the camera 2 or by a device (for example, a drive recorder) in which the camera 2 is mounted.

[0024] Second Embodiment Next, as a second embodiment, an image processing system capable of calculating a distortion correction coefficient for a lens of a camera mounted on a drive recorder from a captured image will be described.

[0025] 2 is a diagram showing an example of the configuration of an image processing system according to the second embodiment. As shown in FIG. Drive recorder 210 is mounted on vehicle 200, and includes camera 211 and flash memory 212. Camera 211 captures an image in the traveling direction of vehicle 200, and the captured image is encoded by a predetermined encoding method and stored as moving image data in flash memory 212. Drive recorder 210 also outputs the moving image data to image processing device 100. Note that camera 211 is an example of camera 2 shown in FIG. 1.

[0026] The image processing device 100 acquires video data from the drive recorder 210. In this embodiment, as an example, the image processing device 100 receives the video data from the drive recorder 210 by communication, but for example, the video data may be acquired via a portable recording medium. The image processing device 100 calculates a distortion correction coefficient for the lens of the camera 211 using the acquired video data. In addition, the image processing device 100 can estimate, for example, the distance to another vehicle captured in the image and the speed of the other vehicle by using the calculated distortion correction coefficient as one of the camera parameters. Such an estimation result of the distance and speed can be used, for example, for analyzing the situation of a traffic accident. The image processing device 100 is an example of the information processing device 1 shown in FIG. 1.

[0027] The image processing device 100 is realized as, for example, a personal computer or a server computer. In this case, the image processing device 100 includes a processor 101, a RAM (Random Access Memory) 102, a HDD (Hard Disk Drive) 103, a GPU (Graphics Processing Unit) 104, an input interface (I / F) 105, a reader 106, and a communication interface (I / F) 107, as shown in FIG.

[0028] The processor 101 performs overall control of the entire image processing device 100. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), a GPU, an application specific integrated circuit (ASIC), or a programmable logic device (PLD). The processor 101 may also be a combination of two or more elements of a CPU, an MPU, a DSP, a GPU, an ASIC, or a PLD.

[0029] The RAM 102 is used as a main storage device of the image processing device 100. The RAM 102 temporarily stores at least a part of an OS (Operating System) program and application programs to be executed by the processor 101. The RAM 102 also stores various data necessary for processing by the processor 101.

[0030] The HDD 103 is used as an auxiliary storage device for the image processing device 100. An OS program, an application program, and various data are stored in the HDD 103. Note that other types of non-volatile storage devices such as an SSD (Solid State Drive) can also be used as the auxiliary storage device.

[0031] A display device 104a is connected to the GPU 104. The GPU 104 displays an image on the display device 104a in accordance with an instruction from the processor 101. The display device 104a may be a liquid crystal display or an organic EL (ElectroLuminescence) display.

[0032] An input device 105a is connected to the input interface 105. The input interface 105 transmits a signal output from the input device 105a to the processor 101. The input device 105a may be a keyboard or a pointing device. The pointing device may be a mouse, a touch panel, a tablet, a touch pad, a trackball, or the like.

[0033] A portable recording medium 106a is detachably attached to the reading device 106. The reading device 106 reads data recorded on the portable recording medium 106a and transmits it to the processor 101. The portable recording medium 106a may be an optical disk, a semiconductor memory, or the like.

[0034] The communication interface 107 transmits and receives data to and from other devices via, for example, a network (not shown). In this embodiment, the video data transmitted from the drive recorder 210 is received by the communication interface 107.

[0035] The processing functions of the image processing device 100 can be realized by the above hardware configuration. As described above, the image processing device 100 calculates the lens distortion correction coefficient of the camera 211 by using the captured image captured by the camera 211. This makes it possible to estimate the lens distortion correction coefficient of the camera 211 even when the model of the drive recorder 210 cannot be identified.

[0036] Here, if the coordinates of a pixel on a captured image are (x, y) and the coordinates of the pixel after distortion correction processing are (x, y), the distortion correction processing is defined by the following equations (1) to (3).

[0037]

number

[0038] Fig. 3 is a diagram showing an example of a checkerboard. As a method for calculating a distortion correction coefficient from a captured image, there is a method using a specific image pattern including feature points whose positions in three-dimensional space are known. As an example of such an image pattern, a checkerboard 30 as shown in Fig. 3 is often used. The checkerboard 30 is photographed by the camera 211, and a distortion correction coefficient is calculated such that the three-dimensional positional relationship of the above-mentioned feature points is restored on the obtained captured image.

[0039] This method of calculating the distortion correction coefficient requires the calculation of the distortion correction coefficient for each camera using a dedicated image pattern, which is time-consuming and inconvenient. In addition, since the calculation result may have errors depending on the shooting method, a certain level of skill is required for the work. For this reason, it is desired to be able to calculate the distortion correction coefficient using an image captured in a general environment without using a dedicated image pattern.

[0040] Fig. 4 shows examples of images before and after distortion correction. Image 31 on the left side of Fig. 4 shows an example of an image in which barrel-shaped radial distortion occurs. Image 32 on the right side of Fig. 4 shows an example of an image in which distortion correction processing has been properly performed on image 31. In image 31 in which radial distortion occurs, lines that actually extend in the vertical or horizontal direction are captured with an inclination relative to the vertical or horizontal direction. Moreover, the inclination differs depending on the position on image 31.

[0041] Images captured in a general environment tend to include many straight line components extending vertically and horizontally. For example, when a road is captured by the drive recorder 210, utility poles and roadside buildings are often captured as subjects, and these subjects include many straight line components extending vertically and horizontally.

[0042] In this embodiment, by utilizing such a property, a distortion correction coefficient is searched for such that many lines extending in the vertical or horizontal direction are included in the image after the distortion correction process, using a photographed image of a general environment. Specifically, the image processing device 100 extracts line segments from the photographed image, performs distortion correction process on each extracted line segment while changing the distortion correction coefficient, and calculates the entropy of the direction histogram for each line segment after the process as an evaluation value.

[0043] If many of the extracted line segments are oriented in a direction close to vertical or horizontal after distortion correction processing, the direction histogram will have prominent frequency peaks at specific angles such as 0 degrees, 90 degrees, and 180 degrees. Since the entropy value decreases as the peak value increases, the image processing device 100 estimates that the distortion correction coefficient that minimizes the entropy value is the optimal distortion correction coefficient.

[0044] FIG. 5 is a diagram showing an example of a captured image with a small number of line segments. The method of calculating the distortion correction coefficient described in FIG. 4 can estimate the distortion correction coefficient with high accuracy when a certain number of line segments or more are extracted from the captured image. However, if the number of line segments extracted from the captured image is too small, there is a problem that the estimation accuracy of the distortion correction coefficient decreases. For example, in a captured image captured by an on-board camera, when a vehicle travels in a place with few utility poles or roadside buildings, there may be cases where the number of straight line components captured in the captured image is extremely small. Only one line segment 41 is extracted from the captured image P1 shown in FIG. 5, and when such a captured image P1 is used, the distortion correction coefficient cannot be estimated with high accuracy.

[0045] Therefore, in this embodiment, the image processing device 100 extracts line segments from each of a plurality of captured images captured at different times using the same camera 211, and estimates a distortion correction coefficient using the extracted line segments. By increasing the number of captured images from which line segments are extracted, the number of line segments used when estimating a distortion correction coefficient can be increased. This makes it possible to accurately estimate a distortion correction coefficient from a captured image even when a location with few utility poles or roadside buildings is captured.

[0046] Fig. 6 is a diagram showing an example of line segment extraction from a plurality of captured images. Fig. 6 shows an example in which line segments are extracted from captured images P1 to Pn captured at times t1, t2, . . . , tn, respectively.

[0047] A line segment 41 is extracted from the photographed image P1. An image P1a is an image showing only the extracted line segment 41 on the photographed image P1. Line segments 42 and 43 are extracted from the photographed image P2. An image P2a is an image showing only the extracted line segments 42 and 43 on the photographed image P2. Line segments 44 and 45 are extracted from the photographed image Pn. An image Pna is an image showing only the extracted line segments 44 and 45 on the photographed image Pn. A line segment composite image P0 is an image in which the extracted line segments 41 to 45 are composited and displayed on the same image.

[0048] It is not necessary to actually generate the images P1a, P2a, Pna and the line segment composite image P0. In practice, it is sufficient to store at least the position information of each of the extracted line segments 41 to 45. As will be described later, in this embodiment, the coordinates of both ends of each of the extracted line segments 41 to 45 are stored.

[0049] Only two or less line segments are extracted from each of the above captured images P1, P2, and Pn. Therefore, if the distortion correction coefficient is estimated using each of the captured images P1, P2, and Pn alone, the distortion correction coefficient cannot be estimated with high accuracy. In contrast, by using the line segments 41 to 45 extracted from each of the captured images P1, P2, and Pn, it is possible to improve the estimation accuracy of the distortion correction coefficient.

[0050] 7 is a diagram showing an example of the configuration of processing functions of the image processing device 100. The image processing device 100 includes a line segment data storage unit 110, an image input unit 121, a line segment extraction unit 122, a line segment addition unit 123, and a distortion correction coefficient estimation unit .

[0051] The line segment data storage unit 110 is a storage area secured in a storage device included in the image processing device 100, such as the RAM 102 or the HDD 103. The line segment data storage unit 110 stores line segment data indicating line segments extracted from a plurality of captured images.

[0052] The processes of the image input unit 121, the line segment extraction unit 122, the line segment addition unit 123, and the distortion correction coefficient estimation unit 124 are realized, for example, by the processor 101 executing a predetermined program.

[0053] Image input unit 121 accepts input of moving image data captured by camera 211. Image input unit 121 extracts captured image data frame by frame from the input moving image data, and inputs the data to line segment extraction unit 122 sequentially.

[0054] The line segment extraction unit 122 extracts line segments from the input photographed image and inputs line segment data indicating each extracted line segment to the image addition unit 123. The line segment adding unit 123 additionally registers line segment data of line segments that do not meet predetermined exclusion conditions among the line segments extracted from the captured image in the line segment data storage unit 110. The exclusion conditions used are that the position of the line segment is included in a predetermined area set near the center of the captured image, and that the position and direction of the line segment are similar to those of line segments stored in the line segment data storage unit 110.

[0055] The distortion correction coefficient estimation unit 124 executes distortion correction processing for each line segment stored in the line segment data storage unit 110 while changing the set value of the distortion correction coefficient, and calculates an evaluation value for the processing result. As the evaluation value, the entropy of the directional histogram for each line segment after the execution of the distortion correction processing is used. The distortion correction coefficient estimation unit 124 estimates an optimal value of the distortion correction coefficient based on the calculated evaluation value.

[0056] Fig. 8 is a diagram showing an example of data stored in the line segment data storage unit. For example, a line segment data table 111 as shown in Fig. 8 is stored in the line segment data storage unit 110. A record for each line segment extracted from a captured image is registered in the line segment data table 111. In each record, a line segment ID for identifying the line segment and endpoint coordinates of one end and the other end of the line segment are registered. The endpoint coordinates indicate coordinates in the captured image before distortion correction processing.

[0057] Next, the processing of the image processing device 100 will be described with reference to a flowchart. FIG. 9 is an example of a flowchart showing the entire distortion correction coefficient calculation process. [Step S11] The image input unit 121 acquires captured images (image frames) from video data. In this step S11, for example, successive captured images in time series are acquired one frame at a time. However, for example, the captured images may be acquired at intervals of a predetermined number of frames.

[0058] [Step S12] The line segment extraction unit 122 extracts line segments from the captured image acquired in step S11 by executing a line segment extraction process on the captured image. The line segment extraction is performed using a known line detection method such as the Hough transform or LSD. The line segment extraction unit 122 temporarily stores the endpoint coordinates of both ends of each extracted line segment in, for example, the RAM 102 as data to be processed in the next step S13.

[0059] [Step S13] The line segment addition unit 123 identifies, from the line segments extracted in step S12, line segments that do not satisfy predetermined exclusion conditions, and executes a line segment addition process to add and register the line segment data of those line segments in the line segment data storage unit 110.

[0060] [Step S14] The line segment adding unit 123 determines whether collection of line segments from the captured images has been completed. For example, when the number of line segments registered in the line segment data storage unit 110 becomes equal to or greater than a predetermined number, it is determined that collection of line segments has been completed. Alternatively, it may be determined that collection of line segments has been completed when the processes of steps S11 to S13 have been executed for all captured images (image frames) included in the video image data. If collection of line segments has not been completed, the process proceeds to step S11, where the next captured image is acquired. On the other hand, if collection of line segments has been completed, the process proceeds to step S15.

[0061] [Step S15] The distortion correction coefficient estimation unit 124 uses the line segments registered in the line segment data storage unit 110 to execute a process of estimating distortion correction coefficients. Fig. 10 is a flowchart showing an example of a line segment addition process. The process in Fig. 10 corresponds to the process in step S13 in Fig. 9. In the following description, the line segment extracted from the captured image in step S12 in Fig. 9 is referred to as a "first line segment," and the line segment registered in the line segment data storage unit 110 is referred to as a "second line segment."

[0062] In addition, if no second line segments are registered in the line segment data storage unit 110 at the start of the processing of Figure 10, all first line segments extracted from the captured image are unconditionally registered in the line segment data storage unit 110.

[0063] [Step S21] The line segment adding unit 123 executes the process loop L1 up to step S30, with each of the first line segments extracted from the captured image as a processing target. [Step S22] The line segment adding unit 123 determines whether the first line segment to be processed is included in a central area set in the center of the captured image. If it is determined that the first line segment is included in the central area, the process proceeds to step S23. If it is determined that the first line segment is not included in the central area, the process proceeds to step S24.

[0064] The central region is a region that is set in advance with the center point of the captured image as the center. For example, the central region is set as a rectangular region having a predetermined number of pixels in the left-right direction and a predetermined number of pixels in the up-down direction from the center point of the captured image. Alternatively, the central region may be set as an elliptical region having a predetermined number of pixels in the left-right direction and a predetermined number of pixels in the up-down direction from the center point of the captured image.

[0065] The first line segment may be determined to be included in the central region when the entirety of the first line segment is included in the central region, or when at least a portion of the first line segment is included in the central region. As an example of the latter, the first line segment may be determined to be included in the central region when more than 50% of the straight line connecting the endpoints of the first line segment is included in the central region.

[0066] [Step S23] The line segment addition unit 123 assigns an addition flag with a value of "0" to the first line segment. [Step S24] The line segment addition unit 123 assigns an addition flag with a value of "1" to the first line segment.

[0067] [Step S25] The line segment adding unit 123 executes the process loop L2 up to step S27, with each of the second line segments registered in the line segment data storage unit 110 as the processing target. [Step S26] The line segment adding unit 123 executes a similar line segment determination process to determine whether a first line segment to be processed in processing loop L1 is similar to a second line segment to be processed in processing loop L2. In this process, if it is determined that the first line segment is similar to the second line segment, the value of the addition flag assigned to the first line segment is updated from "1" to "0".

[0068] [Step S27] When the process of step S26 has been completed for all second line segments registered in the line segment data storage unit 110, the process proceeds to step S28. [Step S28] The line segment addition unit 123 determines whether the value of the addition flag assigned to the first line segment is 1. If the value of the addition flag is 1, the process proceeds to step S29. If the value of the addition flag is 0, the process proceeds to step S30.

[0069] [Step S29] The line segment adding unit 123 adds the first line segment to the line segment data storage unit 110. Specifically, the coordinates of the endpoints of both ends of the first line segment are additionally registered in the line segment data storage unit 110.

[0070] [Step S30] When the processes of steps S22 to S29 for all the first line segments extracted from the captured image have been completed, the line segment addition process ends. 10, among the first line segments extracted from the captured image, the first line segments determined to be included in the central region are not registered in the line data storage unit 110 and are not used in the process of estimating the distortion correction coefficient. Lens distortion is unlikely to occur near the center point of the captured image. Therefore, the shape and direction of the line segments existing near the center point change little before and after the distortion correction process.

[0071] For this reason, the first line segment determined to be included in the central region is considered to have a low contribution to the distortion correction coefficient estimation process and is therefore considered to be of low necessity. By excluding such a first line segment from the distortion correction coefficient estimation process, it is possible to avoid unnecessary calculations in the estimation process without reducing the estimation accuracy, and to shorten the estimation process time.

[0072] Furthermore, in the similar line segment determination process of step S26, if it is determined that the first line segment is similar to any of the second line segments registered in the line segment data storage unit 110, the first line segment is not registered in the line segment data storage unit 110 and is not used in the distortion correction coefficient estimation process. As will be described in the following Fig. 11, a first line segment similar to a second line segment is a first line segment that has an extension direction close to that of the second line segment and is also close to the position of the second line segment. By preventing line segments with similar directions and positions from being used in the distortion correction coefficient estimation process, it is possible to improve the estimation accuracy.

[0073] 11 is a flowchart showing an example of the similar line segment determination process, which corresponds to the process in step S26 in FIG. [Step S41] The line segment adding unit 123 calculates the angle θ between the first line segment to be processed and the second line segment to be processed. In this process, for example, the difference between the angle of a line connecting the endpoints of the first line segment and the angle of a line connecting the endpoints of the second line segment is calculated as the angle θ. Note that the angle of each line is expressed as an angle with respect to a predetermined rotation direction based on a predetermined reference direction (for example, the horizontal direction), and takes a value between 0 degrees and 180 degrees.

[0074] [Step S42] The line segment adding unit 123 determines whether the calculated angle θ is equal to or smaller than a predetermined threshold value. This threshold value is used to determine the closeness of the directions (angles) of the line segments, and an angle greater than but close to 0 is set.

[0075] If the angle θ is equal to or smaller than the threshold, the process proceeds to step S43, and if the angle θ is greater than the threshold, the similar line segment determination process ends. In the latter case, the value of the addition flag added to the first line segment remains at "1," and the first line segment is added to the line segment data storage unit 110 in step S29 of FIG.

[0076] [Step S43] The line segment addition unit 123 calculates the shortest distance d11 from one end point of the first line segment to the second line segment (to the straight line connecting both ends of the second line segment), and the shortest distance d12 from the other end point of the first line segment to the second line segment (to the straight line connecting both ends of the second line segment).

[0077] [Step S44] The line segment addition unit 123 calculates the shortest distance d21 from one end point of the second line segment to the first line segment (to the straight line connecting both ends of the first line segment), and the shortest distance d22 from the other end point of the second line segment to the first line segment (to the straight line connecting both ends of the first line segment).

[0078] [Step S45] The line segment adding unit 123 judges whether at least one of the first and second conditions related to the distance is satisfied. The first condition is that both the distances d11 and d12 are equal to or less than a predetermined threshold, and the second condition is that both the distances d21 and d22 are equal to or less than a predetermined threshold. If at least one of the first and second conditions is satisfied, the process proceeds to step S46, and if neither the first nor the second condition is satisfied, the similar line segment judgment process ends. In the latter case, the value of the addition flag added to the first line segment remains at "1," and the first line segment is added to the line segment data storage unit 110 in step S29 of FIG. 10.

[0079] [Step S46] The line segment adding unit 123 updates the value of the addition flag added to the first line segment to “0.” As a result, the first line segment is no longer added to the line segment data storage unit 110. In the above process of FIG. 11, if a second line segment that is close in both direction and position to a first line segment extracted from a captured image has already been registered in the line segment data storage unit 110, the first line segment is not registered in the line segment data storage unit 110 and is not used in the process of estimating the distortion correction coefficient. Line segments that are close in direction and position change direction in the same way when the distortion correction process is performed. For this reason, if many line segments that are close in direction and position are used in the process of estimating the distortion correction coefficient, there is a possibility that a bias in frequency occurs in a specific angle range (bin) of the direction histogram, and as a result, the estimation accuracy of the distortion correction coefficient may decrease. Therefore, by excluding the above-mentioned first line segment from the target of the process of estimating the distortion correction coefficient, the estimation accuracy can be improved.

[0080] 12 is a flowchart showing an example of a distortion correction coefficient estimation process, which corresponds to step S15 in FIG. In addition, the influence of the circumferential distortion correction coefficients p1 and p2 is small compared to the radial distortion correction coefficients k1, k2, and k3. For this reason, in the following processing, p1 = p2 = 0 is set. Also, to shorten the processing time, k3 = 0 is set, but the optimal value of the distortion correction coefficient k3 may be searched for in addition to the distortion correction coefficients k1 and k2.

[0081] [Step S51] The distortion correction coefficient estimation unit 124 executes the process loop L3 up to step S58 for all combinations of the distortion correction coefficients k1 and k2 that can be set. [Step S52] The distortion correction coefficient estimation unit 124 selects one combination of distortion correction coefficients k1 and k2, and sets the selected distortion correction coefficients k1 and k2 and k3=p1=p2=0 in the above-mentioned equations (1) and (2).

[0082] [Step S53] The distortion correction coefficient estimation unit 124 executes distortion correction processing using equations (1) to (3) for each of the second line segments registered in the line segment data storage unit 110. Specifically, the distortion correction coefficient estimation unit 124 calculates corrected coordinates of each endpoint of the second line segments by converting the coordinates of each endpoint of the second line segments using equations (1) to (3).

[0083] [Step S54] The distortion correction coefficient estimation unit 124 creates a directional histogram for the line segment whose both end point coordinates have been converted by the distortion correction process (the corrected line segment). Specifically, the distortion correction coefficient estimation unit 124 calculates the direction of each of the corrected line segments as the direction (angle) of a straight line connecting the end points of the corrected line segments. Note that the angle of each straight line is expressed as an angle with respect to a predetermined rotation direction based on a predetermined reference direction (e.g., the horizontal direction), and takes a value from 0 degrees to 180 degrees.

[0084] The distortion correction coefficient estimation unit 124 counts the frequency of occurrence for each angle range (bin) obtained by dividing the range from 0 degrees to 180 degrees into l sections based on the angles of the corrected line segments, thereby creating a direction histogram for the corrected line segments.

[0085] [Step S55] The distortion correction coefficient estimation unit 124 calculates a score for evaluating the appropriateness of the set distortion correction coefficients k1 and k2 based on the orientation histogram. The score is calculated using the following equation (4) that calculates the entropy e of the orientation histogram.

[0086]

number

[0087] [Step S56] When step S56 is executed for the first time, the distortion correction coefficient estimation unit 124 temporarily stores the calculated score as the minimum score at that time, for example, in the RAM 102. In this case, execution of the next step S57 is skipped. The set values ​​of the distortion correction coefficients k1 and k2 at this time are also stored in association with the minimum score.

[0088] Furthermore, when step S56 is executed for the second or subsequent times, the distortion correction coefficient estimation unit 124 determines whether the calculated score is smaller than the stored minimum score. If the calculated score is smaller than the minimum score, the process proceeds to step S57. If the calculated score is equal to or greater than the minimum score, the process proceeds to step S58.

[0089] [Step S57] The distortion correction coefficient estimation unit 124 updates the stored minimum score with the calculated score. At this time, the set values ​​of the distortion correction coefficients k1 and k2 at this time are also overwritten and stored in association with the updated minimum score.

[0090] [Step S58] When the processes of steps S52 to S57 have been executed for all combinations of distortion correction coefficients k1 and k2, the process proceeds to step S59. In this case, the minimum value of the scores calculated in step S55 is stored as the minimum score.

[0091] [Step S59] The distortion correction coefficient estimation unit 124 determines the setting values ​​of the distortion correction coefficients k1 and k2 corresponding to the stored minimum score as the optimal values ​​of the distortion correction coefficients k1 and k2, and outputs the optimal values ​​in a form such as storage in the RAM 102 or display on the screen.

[0092] For example, when searching for optimal values ​​of the distortion correction coefficients k1, k2, and k3, the processing loop L3 may be executed for all combinations of the distortion correction coefficients k1, k2, and k3 that can be set. Similarly, when searching for optimal values ​​of the distortion correction coefficients k1, k2, k3, p1, and p2, the processing loop L3 may be executed for all combinations of the distortion correction coefficients k1, k2, k3, p1, and p2 that can be set.

[0093] 12, the distortion correction process is performed on each line segment registered in the line segment data storage unit 110 while changing the combination of the distortion correction coefficients k1 and k2, and an evaluation score (entropy of the orientation histogram) is calculated each time. Then, the distortion correction coefficients k1 and k2 that minimize the calculated score are output as optimal values.

[0094] As mentioned above, the larger the frequency peak in the direction histogram, the smaller the entropy value. In the distortion correction process when the above score is at a minimum, it is considered that a prominent frequency peak occurs at a specific angle such as 0 degrees, 90 degrees, or 180 degrees. In this state, it is considered that many line segments close to the vertical or horizontal direction occur among the line segments after correction, and it can be estimated that the distortion correction process has been performed most appropriately. Therefore, the distortion correction coefficients k1 and k2 set at this time are output as optimal values.

[0095] Then, by using a large number of line segments extracted from multiple captured images and searching for the optimal value of the distortion correction coefficient according to the above procedure, it becomes possible to accurately estimate the distortion correction coefficient even if the captured images include images in which almost no buildings are visible.

[0096] At least a part of the processing functions of the image processing device 100 shown in Fig. 7 may be installed in the drive recorder 210. For example, a processor (not shown) included in the drive recorder 210 may extract lines from a captured image, additionally register the lines in the line data storage unit 110, and estimate a distortion correction coefficient using the registered lines. In this case, the calculated distortion correction coefficient is stored in the flash memory 212 for a predetermined period of time, such as immediately after the drive recorder 210 is attached to the vehicle 200. After that, the image captured by the camera 211 may be corrected using the stored distortion correction coefficient and stored in the flash memory 212.

[0097] In the second embodiment, the distortion correction process is performed on each line segment extracted from a plurality of captured images, and the optimal value of the distortion correction coefficient is calculated using each corrected line segment. However, as another method, a line segment composite image P0 (see FIG. 6) obtained by combining each line segment extracted from a plurality of captured images may be referenced, the distortion correction process may be performed on the line segment composite image P0, line segments may be extracted from the corrected image, and the optimal value of the distortion correction coefficient may be calculated using each extracted line segment. In addition, the luminance gradient of the corrected image may be calculated directly from the line segment composite image P0 using the procedure described in Non-Patent Document 1, and the optimal value of the distortion correction coefficient may be calculated based on the entropy of the directional histogram of the luminance gradient. Furthermore, the procedure described in Patent Document 1 may be used to extract lines from the line segment composite image P0, calculate distortion correction coefficients directly from each extracted line segment, and calculate the final distortion correction coefficient by statistically processing the calculated distortion correction coefficients.

[0098] The processing functions of the devices (for example, the information processing device 1 and the image processing device 100) shown in each of the above embodiments can be realized by a computer. In this case, a program describing the processing contents of the functions that each device should have is provided, and the above processing functions are realized on the computer by executing the program on a computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage devices, optical discs, and semiconductor memories. Examples of magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Examples of optical discs include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

[0099] When distributing a program, for example, the program is recorded on a portable recording medium such as a DVD or CD and then sold. The program can also be stored in a storage device of a server computer and transferred from the server computer to other computers via a network.

[0100] A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from a portable recording medium and execute processing according to the program. The computer can also execute processing according to the received program each time a program is transferred from a server computer connected via a network. [Explanation of symbols]

[0101] 1. Information processing device 1a Processing section 2 Camera 11,12 Photographed images 13 Composite Image 21~23 Line segments X1, X2,..., Xm setting value

Claims

1. The computer A line segment extraction process is executed for each of a plurality of captured images taken at different times to extract a plurality of line segments; executing a distortion correction process while changing a set value of a distortion correction coefficient for the plurality of line segments and evaluating a processing result, thereby calculating an optimal value of the distortion correction coefficient; Distortion correction coefficient calculation method.

2. In the extraction of the plurality of line segments, line segments whose positions and directions are similar to those of other line segments are excluded from the line segments extracted from each of the plurality of captured images, and the remaining line segments are extracted as the plurality of line segments. The distortion correction coefficient calculation method according to claim 1 .

3. In the extraction of the plurality of line segments, line segments included in a predetermined area in the center of the image are excluded from the line segments extracted from each of the plurality of photographed images, and the remaining line segments are extracted as the plurality of line segments. The distortion correction coefficient calculation method according to claim 1 .

4. In the calculation of the optimum value, Each time the distortion correction process is executed while changing the set value, an entropy of a histogram of orientations of the line segments after the distortion correction process is executed is calculated; Calculating the optimal value based on the entropy. The distortion correction coefficient calculation method according to claim 1 .

5. On the computer, A line segment extraction process is executed for each of a plurality of captured images taken at different times to extract a plurality of line segments; executing a distortion correction process while changing a set value of a distortion correction coefficient for the plurality of line segments and evaluating a processing result, thereby calculating an optimal value of the distortion correction coefficient; A distortion correction coefficient calculation program for executing the process.

Images