IMAGE PROCESSING DEVICE AND IMAGE PROCESSING METHOD

Abstract

An image processing device comprises: a straight-line detection unit that detects a straight-line segment from an image captured by a camera; a straight-line tracking unit that temporally tracks the straight-line segment detected from the image by the straight-line detection unit at a first time point and determines an image area containing the straight-line segment from a plurality of divided image areas from an image captured at a second time point later than the first time point; a linearity determination unit that obtains a degree of straightness of the straight-line segment tracked by the straight-line tracking unit at the second time point; and a use-area determination unit that, based on the linearity determined by the linearity determination unit, determines whether the image area containing the straight-line segment is to be used for camera calibration or not.

Description

Technical area

[0001] The present invention relates to an image processing apparatus and an image processing method. State of the art

[0002] In a case where a camera (in-vehicle camera) mounted on a vehicle (automobile) is used as a sensor for automatic driving, high accuracy is required for a mounting position and mounting direction of the in-vehicle camera. However, since the physical mounting accuracy of the in-vehicle camera is limited, a method is used in which the position and angle are accurately measured after the in-vehicle camera is mounted on a vehicle body, and the captured image is corrected based on the measurement result. This is called in-vehicle camera calibration. In-vehicle camera calibration is performed at the time of vehicle shipment, but the mounting angle may shift due to changes over time caused by vehicle vibration or the like.To handle this, a technique for performing calibration while driving is required.

[0003] PTL 1 discloses, as a distortion correction technique for a vehicle camera, a technique for detecting a straight line corresponding to a white line of a road from a captured image and performing correction using the detected white line. Literature listPatent literature

[0004] PTL 1: JP 2011-185753 A Summary of the inventionTechnical problem

[0005] Meanwhile, a camera mounted with a wide-angle lens for wide-area detection is often used as the vehicle camera, and the image is converted into an image with wide-angle distortion corrected at the time of calibration. However, even though the wide-angle distortion is corrected, the distortion in the peripheral portion of the image captured by the wide-angle lens is large, causing an error in the calibration for obtaining the camera's direction.

[0006] In addition, when correcting wide-angle distortion, there is a problem that the distortion correction error changes due to environmental changes such as temperature changes. Therefore, the range usable for calibration is not fixed and may change during driving.

[0007] However, it is not possible to solve such a problem without changing the distortion correction error by the conventional distortion correction technique disclosed in PTL 1.

[0008] In this respect, it is an object of the present invention to provide an image processing apparatus and an image processing method capable of performing appropriate image calibration even in a case where an environment changes, for example, during driving. Solution to the problem

[0009] In order to solve the above problems, configurations described in the claims, for example, are used.

[0010] The present application provides a variety of means for solving the above problems. An example of the image processing device includes: a straight line detection unit that detects a straight line portion from an image captured by a camera; a straight line tracking unit that sequentially tracks the straight line portion detected by the straight line detection unit from the image at a first time point and determines an image region including the straight line portion from a plurality of divided image regions from an image captured at a second time point later than the first time point; a linearity determination unit that obtains a degree of straightness of the straight line portion tracked by the straight line tracking unit at the second time point.and a range determination unit that determines whether or not to use the image range including the straight line portion for camera calibration based on the degree of straightness determined by the linearity determination unit; Advantageous effects of the invention

[0011] According to the present invention, even in a case where a distortion correction error of an image has occurred due to an environmental change such as temperature, calibration is performed using only a region with less distortion, and thus it is possible to prevent an error due to calibration from occurring and to maintain a correct calibration state.

[0012] Objects, configurations, and advantageous effects other than those described above will be clarified by the descriptions of the following embodiments. Brief description of the drawings Fig. 1 is a diagram illustrating an example of a vehicle configuration and a hardware configuration of an image processing apparatus according to a first embodiment of the present invention. [ Fig. 2] Fig. 2 is a configuration diagram of the image processing apparatus according to the first embodiment of the present invention. [ Fig. 3] Fig. 3 is a flowchart illustrating an example of image processing according to the first embodiment of the present invention. [ Fig. 4] Fig. 4 is a diagram illustrating a division example (Example 1) of an image area according to the first embodiment of the present invention. [ Fig. 5] Fig. 5 is a diagram illustrating an example of tracking a straight line section according to the first embodiment of the present invention. [ Fig. 6] Fig. 6 is a diagram illustrating a division example (Example 2) of the image area according to the first embodiment of the present invention. [ Fig. 7] Fig. 7 is a diagram illustrating a division example (Example 3) of the image area according to the first embodiment of the present invention. [ Fig. 8] Fig. 8 is a configuration diagram of an image processing apparatus according to a second embodiment of the present invention. [ Fig. 9] Fig. 9 is a configuration diagram of an image processing apparatus according to a third embodiment of the present invention. Description of the embodiments<Erstes Ausführungsbeispiel>

[0013] A first embodiment of the present invention will be described below with reference to the Fig. 1 to Fig. 7 described.

[0014] Fig. 1 illustrates a configuration of a vehicle 1 on which an image processing apparatus of the present embodiment is mounted.

[0015] A vehicle camera 2 is installed in the vehicle (car) 1 to monitor the front. Although Fig. 1 illustrates two vehicle cameras 2 for front monitoring, the number and positions of the vehicle cameras 2 are not limited to the example of Fig. 1, and only one vehicle camera 2 may be provided. Furthermore, it should be noted that in a case where two vehicle cameras 2 are provided as in Fig. 1, a stereo camera described in another embodiment described later may be provided.

[0016] An image captured by the vehicle camera 2 is supplied to an image processing device 3 in an information processing device 5.

[0017] In the image processing device 3, image processing of a captured image and image correction processing, such as camera calibration and distortion correction, are performed. Then, an object, a person, or the like is detected from an image subjected to image correction processing. Note that when detecting an object, a person, or the like, not only an image captured by the in-vehicle camera 2 but also another detection mechanism, such as a radar (not illustrated), may be used in combination.

[0018] As in Fig. 1, the information processing apparatus 5 in which the image processing apparatus 3 is installed includes a central processing unit (CPU) 5a, a memory 5b, a storage unit 5c, and an interface (I / F) 5d, and these are connected to be able to transfer data to each other.

[0019] The CPU 5a causes a program stored in the storage unit 5c to be executed in the working memory 5b, thereby causing the working memory 5b to configure each processing unit that performs image processing.

[0020] The storage unit 5c stores programs for performing image processing and vehicle control, and stores images, control data, and the like.

[0021] The interface 5d performs input processing of image data from the vehicle camera 2 and detection data from various sensors and output processing of control data of the vehicle.

[0022] Information of a result of recognition of an object or a person by the image processing device 3 configured as described above is transmitted to a vehicle control device 4. The vehicle control device 4 acquires a recognition result of an object or a person as information necessary for automatic driving or the like, and controls a driving state of the vehicle 1. For example, in the vehicle control device 4, a braking operation for avoiding an object or a person in front of the vehicle 1, a control for adjusting a speed to follow another vehicle in front of the vehicle 1, a steering operation for performing traveling while maintaining a lane of a traveling road, and the like are executed.

[0023] In a case where the in-vehicle camera 2 is used for vehicle control such as automatic driving, it is necessary to know exactly which direction the in-vehicle camera 2 is facing. The direction of the camera 2 is determined at a time when the in-vehicle camera 2 is mounted on the vehicle 1, but since an individual difference occurs depending on the vehicle 1 or the camera 2, the accuracy becomes insufficient if the determination value is used as it is. Therefore, it is necessary to measure the direction of the camera 2 for each vehicle after the camera 2 is mounted on the vehicle 1. This is called external parameter calibration.Generally, external parameter calibration is performed when the vehicle 1 is shipped from a factory. However, vibration or the like occurs during operation of the vehicle 1, and thus, it is necessary to perform calibration periodically, taking long-term operation into account. This method of automatically performing calibration while the vehicle 1 is running may be referred to as calibration-while-running or calibration-on-the-fly.

[0024] Calibration during driving is performed by tracking a characteristic object appearing in the in-vehicle camera 2. However, since the in-vehicle camera 2 needs to detect a wide area around the vehicle, a wide-angle camera with an imaging lens with lens distortion is used. Note that if the lens characteristics are measured in advance, it is possible to convert to an image without distortion. Therefore, the characteristics related to lens distortion are also measured before the camera is shipped.

[0025] A distortion correction parameter is used to remove lens distortion at the time of calibrating external parameters during driving, but due to the characteristics of the wide-angle camera, the original distortion in the peripheral portion is larger than in the central portion of the image, so a correction error is likely to occur.

[0026] In addition, the environment during actual operation exhibits large temperature changes and the like compared to the stable environment in the factory. Due to the influence of a temperature or the like different from that in the factory, the distortion state changes and the correction error increases. Therefore, when using an image of the peripheral section, a large error may occur, adversely affecting the calibration of external parameters.

[0027] Here, in the present embodiment, the correction error of the lens distortion is measured during driving, and only a region with less distortion is used for calibration during driving.

[0028] Fig. 2 is a configuration diagram of the image processing device 3 and illustrates a configuration for the image processing device 3 to determine a range to be used for calibration when calibrating an image captured by the vehicle camera 2.

[0029] The image processing device 3 includes an image acquisition unit 11, a distortion removal unit 12, a straight line detection unit 13, a straight line tracking unit 14, a linearity determination unit 15, a region-specific suitability determination unit 16, a region to be used determination unit 17, an image region limiting unit 18, and an external parameter estimation unit 19.

[0030] The image capture unit 11 captures an image taken by the vehicle camera 2.

[0031] The distortion removal unit 12 removes distortion from the image captured by the image acquisition unit 11. For example, the distortion removal unit 12 removes lens distortion that occurs in the peripheral portion of an image captured by a wide-angle lens. A distortion removal process in the distortion removal unit 12 is performed using predetermined parameters.

[0032] The straight line detection unit 13 detects a straight line structural object, which is captured as an image, from the captured image from which the lens distortion has been removed by the distortion removal unit 12. In this example, the straight line detection unit 13 detects a straight line structural object in the image. Here, the straight line structural object is a structural section including a straight line section in a substantially vertical direction, such as a utility pole or the end of a building.

[0033] The straight line tracking unit 14 tracks the location of the straight line structure object detected by the straight line detection unit 13 in an image captured at a later time. That is, as long as the vehicle 1 is moving, the area in which the vehicle camera 2 performs image capture changes every frame. Thus, the straight line tracking unit 14 tracks the position of the same straight line structure section in each frame.

[0034] The linearity determination unit 15 determines a linearity of a straight-line structural object in the image tracked by the straight-line tracking unit 14. When the linearity determination unit 15 determines the linearity in a state where the captured image is divided into a plurality of regions, the region-specific suitability determination unit 16 determines a region in which the linearity determination was performed and obtains a determination result of the suitability of the linearity for each region. An example of dividing an image into a plurality of regions will be described later ( Fig. 4 and the like).

[0035] The area-to-use determination unit 17 determines the area of ​​the image to be used for calibration from the determination result of the suitability of linearity for each area in the area-specific suitability determination unit 16.

[0036] The image area limiting unit 18 extracts only image information of the area determined by the area-to-use determining unit 17 as being used for calibration from an image in which the distortion removing unit 12 has removed distortion. The image information of the area extracted by the image area limiting unit 18 is supplied to the external parameter estimating unit 19. The external parameter estimating unit 19 estimates a parameter for performing image calibration. The parameter estimation performed by the external parameter estimating unit 19 is for performing calibration during driving. However, the external parameter estimating unit 19 may perform parameter estimation in addition to the driving time. In addition, the driving time here may be a time during which the vehicle is actually traveling or a time during which the vehicle is temporarily stopped during traveling.

[0037] As the parameter estimation performed by the external parameter estimation unit 19, parameter estimation by various methods conventionally known as camera calibration may be applied.

[0038] For example, the external parameter estimation unit 19 detects a feature point on a white line of a road from the image and determines a coordinate position of a point at which extensions of a plurality of white lines indicated by feature points intersect as a road vanishing point.

[0039] Then, the external parameter estimation unit 19 calculates the installation height and angle of the in-vehicle camera 2 from the coordinate position information. Furthermore, if the calculated installation height and angle of the in-vehicle camera 2 are shifted from the initial set installation height and angle of the in-vehicle camera 2, the external parameter estimation unit 19 corrects the initial set installation height and angle. A camera calibration process by parameter estimation described here is an example, and other methods may be used. However, only the area determined by the area determination unit 17 to be used is used as the image used for parameter estimation.

[0040] Fig. 3 is a flowchart illustrating an example of an operation in which the image processing device 3 in the present example performs the external parameter calibration during driving, which is carried out with the method shown in Fig. 2 illustrated configuration.

[0041] First, the image captured by the in-vehicle camera 2 is captured by the video acquisition unit 11 and converted into a distortion-free image by the distortion removal unit 12. A parameter obtained by measurement in a vehicle manufacturing factory or the like is used as the conversion parameter in the distortion removal unit 12.

[0042] The image subjected to distortion removal by the distortion removal unit 12 is transmitted to the straight line detection unit 13. The straight line detection unit 13 performs a straight line detection process to detect a linear target object in an area with a small distortion correction residual near the center of the image (step S11). Here, the area near the center where the linear target object is detected is based on a predefined range.

[0043] Then, the straight line tracking unit 14 performs a straight line tracking process in which a linear object detected from one frame at a time when the straight line detection process is executed in step S11 is tracked over a plurality of frames until the linear object comes out of the screen due to the movement of the vehicle 1 (step S12).

[0044] Thereafter, the linearity determination unit 15 executes a linearity determination process for determining a linearity at a position where the linear structure object tracked by the straight line tracking unit 14 has moved in the image (step S13).

[0045] The determination result in step S13 is accumulated in the area-specific suitability determination unit 16 as a linearity suitability result for each of a plurality of areas obtained by dividing the image. The accumulation of the linearity suitability results in the area-specific suitability determination unit 16 is performed for at least a predetermined time. Then, the area-to-be-used determination unit 17 determines, based on the determination result for each area in the area-specific suitability determination unit 16, whether or not each area is usable for calibration (step S14).

[0046] After the determination in step S14, the image range limiting unit 18 extracts the captured image of the range determined by the to-be-used range determining unit 17 as usable for calibration. Then, the external parameter estimation unit 19 performs an external parameter estimation process for calibration using the captured image of the range extracted by the image range limiting unit 18 (step S15). Note that in a case where it is determined in step S14 that almost no range is usable for calibration (for example, in a case where it is determined that linearity is maintained only near the center), the image range limiting unit 18 does not extract an image for calibration.

[0047] After the calibration process during driving is executed in step S15, the image processing device 3 executes the processing of step S11 at a time of executing the next calibration process.

[0048] Furthermore, it should be noted that the calibration process during driving is performed, for example, every predetermined time from the start of driving (engine start). Alternatively, in a case where an environment such as temperature changes by a predetermined value during driving, the calibration process is performed during driving. Alternatively, based on performing calibration every predetermined time in addition to this, calibration may also be performed in a case where the temperature changes by a predetermined value.

[0049] Fig. 4 illustrates a division example when the area-specific suitability determination unit 16 divides the captured image into a plurality of areas.

[0050] In the example of Fig. 4, an image of a single frame is divided into a plurality of stripes. That is, the image is divided into a plurality of regions at substantially predetermined intervals by lines in the vertical direction, and 10 regions a1-1, a1-2, a1-3, a1-4, a1-5, a1-6, a1-7, a1-8, a1-9, and a1-10 are set. The region-specific suitability determination unit 16 accumulates determination results of the suitability of linearity for a predetermined period of time in each of the 10 regions a1-1 to a1-10.

[0051] Fig. 5 is a diagram for describing details of the processing in the linearity determination unit 15.

[0052] The linearity determination unit 15 first detects linear feature objects (linear objects) X1 and X2 at a position of the central portion Pa of an image P1 on the left side of Fig. 5. Here, it is assumed that the feature objects X1 and X2 are, for example, utility poles on the left and right side of a road surface R.

[0053] For detecting the linear feature objects X1 and X2, for example, a Hough transform, which is known as a method for detecting an object with an element of a straight line or a circle from an image, can be used.

[0054] A linear approximation is performed on the feature object detected by the Hough transform and a linearity is determined from a distance from the straight line to each point.

[0055] The linear objects X1 and X2 are tracked over a plurality of frames, and the linearity determination unit 15 determines the linearity of the feature objects X1 and X2 again at the tracked location. For example, the linearity of the tracked feature objects X1 and X2 is determined from a captured image P2 (the image on the right in Fig. 5) after a predetermined time from the image P1.

[0056] In the determination method of linearity here, as in the case where the linearity is first detected from the image P1, an approximately straight line is obtained and the linearity is determined by a distance from each point of an object to the proximity sample.

[0057] Then, when the linearity is within a predetermined threshold, the linearity determination unit 15 determines that the line at the position is straight and determines that the linearity is maintained in a range including that position.

[0058] The divided area Pb ( Fig. 5 illustrates only the left end area), which is determined here are the areas a1-1 to a1-10, which are divided by vertical lines, as in Fig. 4 illustrates.

[0059] Furthermore, in a case where the linearity is not maintained beyond the threshold, the linearity determination unit 15 determines that the linearity is not maintained at that position. By repeating this determination, the linearity determination unit 15 determines the presence or absence of linearity for a plurality of captured objects. After a predetermined number of determinations are made or after a predetermined time has elapsed, the area-specific suitability determination unit 16 aggregates the determination results of the suitability of the linearity for each area and determines the suitability for each area.That is, in the area-specific suitability determination unit 16, an area with a high ratio in which linearity is maintained is determined as an allowable area, and an area with a high ratio in which linearity is not maintained is determined as an inadmissible area.

[0060] The use area determination unit 17 sets the area determined by the area-specific suitability determination unit 16 as a use area and determines only the area as a usable area for calibration during driving. Thus, it is possible to perform appropriate calibration during driving based on the determination of the linearity of an object tracked during actual driving of the vehicle.

[0061] Using the captured image of the area determined by the area-to-be-used determination unit 17, the external parameter estimation unit 19 executes the external parameter estimation process for calibration during driving.

[0062] As described above, according to the present embodiment, an object originally appearing as a straight line is tracked, and it is confirmed that linearity is maintained at various locations on the screen. As a result, the area where linearity is not maintained is determined as the area with a large distortion correction error, and this area is excluded at the time of on-the-fly calibration, so that it is possible to perform on-the-fly calibration with high accuracy by using only the area where the distortion correction error is small.Thus, even in a case where a distortion correction error of an image has occurred due to a temperature change during driving or the like, it is possible to prevent the occurrence of an error due to calibration and maintain correct calibration by using only an area with less distortion.

[0063] In addition, in a case where the usable range decreases, erroneous automatic calibration can be prevented by stopping the calibration.

[0064] That is, in a case where a straight line in the Fig. 2 illustrated configuration and processing of the flowchart of Fig. 3 is tracked, and there are few (almost no) areas where it can be determined that linearity is maintained, calibration is essentially not performed, and erroneous automatic calibration can be prevented. For example, in a situation where the temperature of the vehicle in which the camera is installed is temporarily very high and distortion is large, it is possible to prevent an error from occurring due to calibration at the time of the temporary high temperature.

[0065] It should be noted that the setting of the ranges a1-1 to a1-10, which are determined by the Fig. 4 illustrated vertical lines, an example is and other area divisions than the vertical lines, as in Fig. 6 or Fig. 7 illustrates, can be used.

[0066] The Fig. 6 is an example in which an image pickup area A2 is concentrically divided from an optical axis center of an imaging lens to set areas a2-1, a2-2, a2-3, a2-4, a2-5, and a2-6.

[0067] Since the distortion due to the wide-angle lens is determined by a distance from the optical axis center, the lens distortion correction error is expected to be determined by the distance from the optical axis center. Therefore, it can be expected that distortion correction residuals in the same area become equivalent by using the area division in the Fig. 6 illustrated concentric form.

[0068] The Fig. The example illustrated in Figure 7 is an example in which an image pickup area A3 is divided into rectangular areas a3-1, a3-2, a3-3, a3-4, and a3-5 centered on the optical axis center of the imaging lens. When determining in which of the areas a3-1 to a3-5 a linear object is present, the calculation amount can be expected to be smaller than that of the circular area division, as shown in Fig. 6 illustrates.

[0069] It should be noted that one of the examples of Fig. 4, Fig. 6 and Fig. 7 can be applied to the area subdivision, but for example in a case where the area is defined by the Fig. 4 illustrated vertical lines, the area can also be determined only by the lateral position (horizontal position) of the object, so that it is possible to perform the area determination by the simplest calculation.

[0070] In addition, the temporal change in lens distortion can also be determined by temperature measurement, as physical deformation due to temperature change is a major cause. In this case, the usable range is determined in advance according to the temperature, and the usable range during driving is determined by the currently measured temperature. In addition, by additionally using the measured temperature, it is also possible to determine the range based on temperature information only in cases where the number of detected linear objects is small or detection is not possible. <Zweites Ausführungsbeispiel>

[0071] Next, a second embodiment of the present invention will be described with reference to Fig. 8 described. In Fig. 8, which illustrates the second embodiment, the same reference numerals are applied to portions corresponding to the Fig. 1 to Fig. 7, which are described in the first embodiment.

[0072] In the second embodiment of the present invention, two vehicle cameras 2L and 2R are prepared as a stereo camera, and calibration of the vehicle cameras 2L and 2R as the stereo camera is performed. That is, as shown in Fig. 8, the vehicle camera 2L that obtains a left image by image pickup and the vehicle camera 2R that obtains a right image by image pickup are mounted on the vehicle.

[0073] Then, as in Fig. 8 illustrates, the captured images of the vehicle cameras 2L and 2R are captured by the image capturing units 11a and 11b, and the distortion removing units 12a and 12b remove lens distortion from the captured images.

[0074] After that, only the distortion-removed image of the left image is fed to the straight line detection unit 13. After that, similar to the configuration of Fig. 2, the straight line tracking unit 14, the linearity determination unit 15, the area-specific suitability determination unit 16, and the area-to-be-used determination unit 17 sequentially perform processing to determine the area of ​​use based on the determination result of the tracked linearity. Details of the processing performed from the straight line detection unit 13 to the area-to-be-used determination unit 17 are the same as those shown in the flowchart of Fig. 3 described processing.

[0075] Then, the distortion-removed image of the left image obtained by the distortion removal unit 12a and the distortion-removed image of the right image obtained by the distortion removal unit 12b are supplied to the image region limiting unit 18. The image region limiting unit 18 extracts only the use region instructed by the use region determining unit 17 from the left image and the right image, and the external parameter estimation unit 19 performs the external parameter estimation process for calibration.

[0076] By performing the processing with the Fig. With the configuration illustrated in Figure 8, it is possible to perform calibration while driving of the in-vehicle cameras 2L and 2R, which are the stereo camera. Furthermore, although two cameras are arranged, the configuration and processing from the straight line detection unit 13 to the area determination unit 17 to be used can be established based only on the captured image of one camera, simplifying the configuration and processing as the image processing device.

[0077] It should be noted that in this case the determination of the area of ​​use is only carried out for the left image as an example and the determination of the area of ​​use can equally be carried out only for the right image. <Drittes Ausführungsbeispiel>

[0078] Next, a third embodiment of the present invention will be described with reference to Fig. 9. In Fig. 9, which illustrates the third embodiment, the same reference numerals are assigned to portions corresponding to the Fig. 1 to Fig. 8 described in the first and second embodiments.

[0079] In the third embodiment of the present invention, as in the second embodiment, two vehicle cameras 2L and 2R are prepared as a stereo camera, and calibration of the vehicle cameras 2L and 2R as the stereo camera is performed. Fig. 8, the captured images of the vehicle cameras 2L and 2R are captured by the image capturing units 11a and 11b, and the distortion removing units 12a and 12b remove lens distortion from the captured images.

[0080] Then, in the present embodiment, the distortion-removed image of the left image is supplied to the straight line detection unit 13a, and then processing is performed for the left image sequentially by the straight line tracking unit 14a, the linearity determination unit 15a, and the area-specific suitability determination unit 16a. Furthermore, the distortion-removed image of the right image is supplied to the straight line detection unit 13b, and then determination processing is performed for the right image by the straight line tracking unit 14b, the linearity determination unit 15b, and the area-specific suitability determination unit 16b.

[0081] Then, the area-specific suitability determination result of the left image obtained by the area-specific suitability determination unit 16a and the area-specific suitability determination result of the right image obtained by the area-specific suitability determination unit 16b are transmitted to an area determination unit 17 to be used.

[0082] The area-to-use determination unit 17 comprehensively determines the area-specific suitability determination result of the left image and the area-specific suitability determination result of the right image, and determines the use area for calibration.

[0083] Based on the determination result in the area determination unit 17 to be used, the image area restriction unit 18 extracts specific areas of the left image and the right image, transmits the areas to the external parameter estimation unit 19, and causes the external parameter estimation unit to execute the external parameter estimation process for calibration.

[0084] By performing the processing with the Fig. In the configuration illustrated in Figure 9, it is also possible to perform calibration while driving of the vehicle cameras 2L and 2R, which are the stereo camera.

[0085] Here, in the case of the present embodiment, since the area-specific suitability determination process is executed for both the left image and the right image, optimal area-specific suitability determination is performed for each of the vehicle cameras 2L and 2R, and more accurate calibration of the stereo camera during driving can be performed. <modifikationsbeispiele>

[0086] It should be noted that the embodiments described here have been described in detail in order to explain the present invention in an easy-to-understand manner, and the above embodiments are not necessarily limited to a case including all of the described configurations. For example, in each of the embodiments described above, a straight-line structural object is tracked in an image captured during driving, and calibration is performed during driving. However, the detection that linearity is maintained may be performed during driving, and the calibration itself may be performed during a temporary stop at the time of driving or the like. Furthermore, the point of performing calibration during driving is merely an example, and the determination and calibration may be performed at other times.

[0087] In addition, the Fig. 1, Fig. 2, Fig. 8 and Fig. While the configuration diagrams, command lines, and information lines considered necessary for the description are illustrated in Figure 9, not all command lines and information lines ultimately integrated into a product need necessarily be illustrated. In practice, it can be assumed that almost all components are interconnected.

[0088] In addition, the operational processing of the Fig. The flowchart illustrated in Figure 3 is also intended as an example. As long as the processing results are the same, some of the processing order can be changed, or multiple processes can be executed simultaneously.

[0089] Furthermore, the processing performed in the image processing device in the present invention is performed by executing a program (software). For example, a known information processing device can be made to function as the image processing device in the present invention by mounting a program that executes the processing described in each embodiment. Information of the program in this case can be stored in various recording media, such as a memory, a hard disk, a solid-state drive (SSD), an IC card, and an SD optical disk.

[0090] In addition, in the Fig. 1, the image processing device is configured by an information processing device (a computer) executing under the control of the CPU, however, some or all of the functions performed by the image processing device may be realized by dedicated hardware such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). List of reference symbols 1 vehicle 2 vehicle cameras 2L, 2R vehicle camera (stereo camera) 3 Image processing device 4 Vehicle control device 5 Information processing device 5a CPU 5b RAM 5c storage unit 5d interface 11, 11a, 11b Image acquisition unit 12, 12a, 12b Distortion removal unit 13, 13a, 13b Straight line detection unit 14, 14a, 14b Straight line tracking unit 15, 15a, 15b Linearity determination unit 16, 16a, 16b area-specific suitability determination unit 17 area determination unit to be used 18 Image area limitation unit 19 external parameter estimation unit QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2011-185753 A

[0004] < / modifikationsbeispiele>

Claims

[1] An image processing apparatus comprising: a straight line detection unit that detects a straight line section from an image captured by a camera; a straight line tracking unit that sequentially tracks the straight line portion detected from the image by the straight line detection unit at a first time point and determines an image area including the straight line portion from a plurality of divided image areas from an image captured at a second time point later than the first time point; a linearity determination unit that obtains a degree of straightness of the straight line section tracked by the straight line tracking unit at the second time point; and a range determination unit to be used, which determines whether or not the image range including the straight line portion is to be used for camera calibration based on the degree of straightness determined by the linearity determination unit. [2] The image processing apparatus according to claim 1, wherein, when the straight line tracking unit time-sequentially tracks a straight line portion detected in an image area near a center of an image among the plurality of divided image areas and detects that a straight line portion detected in an image area of a peripheral portion of the image has moved, the area to be used determining unit determines whether or not the image area of the peripheral portion is usable for calibration of the camera from the determination of the degree of straightness in the corresponding image area by the linearity determining unit. [3] The image processing device according to claim 2, wherein the calibration of the camera is performed by using only an area determined as usable for the calibration of the camera determined by the area-to-be-used determining unit while a vehicle on which the image processing device is mounted is traveling or while the vehicle is temporarily stopped during traveling. [4] The image processing apparatus according to claim 3, wherein the image area determined by the straight line tracking unit is an image area in which the image picked up by the camera is divided by arranging dividing lines extending in a vertical direction at predetermined intervals in a horizontal direction. [5] The image processing apparatus according to claim 1, wherein the camera is a stereo camera that includes a left camera that receives a left image and a right camera that receives a right image, the straight line detection unit, the straight line tracking unit, the linearity determination unit and the area to be used determination unit perform processing on a captured image of the left camera or the right camera, and the area determined by the area determination unit to be used as being used for calibration is applied to both a captured image of the left camera and a captured image of the right camera. [6] The image processing apparatus according to claim 1, wherein the camera is a stereo camera that includes a left camera that receives a left image and a right camera that receives a right image, the straight line detection unit, the straight line tracking unit and the linearity determination unit each perform processing on a captured image of the left camera and a captured image of the right camera, Degree determination results of the straightness of the captured image of the left camera are accumulated for each area and degree determination results of the straightness of the captured image of the right camera are accumulated for each area, and the area determination unit to be used determines an area to be used for calibration based on area-specific accumulation information for the captured image of the left camera and area-specific accumulation information for the captured image of the right camera, and applies the determined area to both the captured image of the left camera and the captured image of the right camera. [7] An image processing method for calibrating, with an image processing device installed in a vehicle, an image taken by a camera mounted on the vehicle, the image processing method causing the image processing device to perform the following: a straight line detecting process for detecting a straight line section from the image captured by the camera; a straight line tracking process for temporally sequentially tracking a straight line portion detected by the straight line detection process at a first time point and determining an image region including the straight line portion from image regions obtained by dividing an image captured at a second time point later than the first time point; a linearity determination process for obtaining a linearity of the straight line section tracked by the straight line tracking process at the second time point; and a range determination process to be used for determining whether or not the image range including the straight line portion is used for calibrating the camera based on the linearity determined by the linearity determination process.

Images