Image processing apparatus and camera parameter change detection method

Abstract

To detect changes in more camera parameters.SOLUTION: An image acquisition unit 101 that acquires a plurality of images captured by a plurality of cameras 11 having different viewpoints, a parallax calculation unit 102 that obtains parallax information from the plurality of images, a plane estimation unit 103 that estimates a plane region present in an external world captured in the plurality of images on the basis of the parallax information, and a collation unit that collates a converted image obtained by converting one image among the plurality of images using an estimation result of the plane region with another image, A parameter change detection part 104 for detecting that at least one of a plurality of internal parameters and a plurality of external parameters of the camera and a distortion parameter of a lens is changed on the basis of a result of the collation, and an output part 105 for outputting that the camera parameter is changed when the change of the camera parameter is detected by the parameter change detection part are provided.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device that processes captured images. [Background technology]

[0002] In recent years, the widespread use of in-vehicle camera systems that recognize objects based on images captured by cameras has led to increased demand for various recognition functions aimed at safer and more autonomous driving. Among these, stereo camera systems that perform object detection using two cameras arranged side by side can measure not only visual information from images but also distance information to objects, allowing for detailed understanding of various objects around the vehicle (people, cars, three-dimensional objects, road surfaces, road signs, signboards, etc.), and are said to contribute to improved safety during driving assistance.

[0003] The camera parameters of an in-vehicle camera, such as the camera's attitude and lens distortion characteristics, change due to vibrations that occur when the vehicle is traveling, changes in the temperature of the environment in which it is used, etc. For example, in a stereo camera, the distance from the camera to an object is estimated from the difference (parallax) between the positions of the same object captured by the left and right cameras, but changes in the camera parameters can cause errors between the estimated distance and the actual distance, which can hinder driving assistance.

[0004] A technology for detecting changes in camera parameters of cameras constituting a stereo camera from images of the cameras is disclosed in, for example, Patent Document 1. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-113434 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the technology disclosed in Patent Document 1, the camera parameters for which changes can be detected are limited to the relative pitch angle and roll angle of the comparison camera with respect to the reference camera, and changes in other camera parameters are not taken into consideration.

[0007] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an image processing device that can detect changes in a greater number of camera parameters when the camera parameters of multiple cameras used to capture stereo images change. [Means for solving the problem]

[0008] In one preferred aspect, the image processing device according to the present invention comprises an image acquisition unit that acquires multiple images captured by multiple cameras with different viewpoints; a parallax calculation unit that calculates parallax information from the multiple images; a plane estimation unit that estimates a planar area existing in the external world captured in the multiple images based on the parallax information; a parameter change detection unit that compares a converted image obtained by converting one of the multiple images using the estimated planar area with other images, and detects, based on the comparison result, that a change has occurred in at least one of multiple internal parameters of the camera, multiple external parameters, and lens distortion parameters; and an output unit that, when a change is detected by the parameter change detection unit, outputs information that the camera parameters have changed. [Effects of the Invention]

[0009] According to the present invention, it is possible to detect changes in a greater number of camera parameters for multiple cameras used to capture stereo images, thereby increasing opportunities to prompt readjustment of the camera parameters, and preventing a decrease in the accuracy of estimating the distance to the object, etc. Other novel features of the present invention and the technical problems solved by these novel features will become apparent from the description and drawings of this specification. [Brief explanation of the drawings]

[0010] [Figure 1]1 is a schematic configuration diagram of an image processing device according to a first embodiment of the present invention. [Figure 2] FIG. 1 is a simplified block diagram showing a hardware configuration of an image processing device. [Figure 3] 10 is a flowchart of a detection process performed by the image processing device. [Figure 4] FIG. 10 is a schematic configuration diagram of an image processing device according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] FIG. 1 is a schematic block diagram showing the configuration of an image processing apparatus according to the first embodiment of the present invention.

[0013] The image processing device 1 includes image acquisition units 101 - a and 101 - b , a parallax calculation unit 102 , a plane estimation unit 103 , a first parameter change detection unit 104 , and an output unit 105 .

[0014] Image acquisition units 101-a and 101-b acquire images from cameras 11-a and 11-b mounted on the vehicle. Cameras 11-a and 11-b may be general stereo cameras with a fixed relative positional relationship with a predetermined baseline length, or may be independent cameras fixed at different positions within the vehicle. In this case, a non-parallel stereo camera may be configured in which the optical axes of the two cameras are non-parallel. While this embodiment is described assuming that camera 11 is two general stereo cameras, three or more cameras may be installed on the vehicle, and two cameras with overlapping fields of view may be used to detect objects in multiple directions. Hereinafter, in this specification, when multiple components have the same function and / or configuration, they will be referred to by a common numerical reference number followed by an alphabetical suffix. When there is no need to particularly identify individual components, the alphabetical suffix will be omitted, e.g., image acquisition unit 101.

[0015] The parallax calculation unit 102 combines an image acquired by the image acquisition unit 101-a from a reference camera, for example, camera 11-a, with an image acquired by the image acquisition unit 101-b from a comparison camera, for example, camera 11-b, based on the shooting date and time, to generate parallax information. The parallax information can be generated by associating identical objects in the real world captured by the two cameras and deriving the difference in coordinates of the same point in each image. Note that image matching methods such as SGM (Semi Global Matching) and SAD (Sum of Absolute Difference) can be used to generate parallax information using multiple images captured by multiple cameras.

[0016] The plane estimation unit 103 estimates a planar area existing in the external world captured in the multiple camera images based on the parallax information generated by the parallax calculation unit 102. Here, a planar area refers to an area that is recognized as a continuous area whose horizontal distance from the camera is constant or changes with a constant gradient.

[0017] The first parameter change detection unit 104 compares an image captured by one of the multiple cameras, for example, camera 11-b that is used for comparison in the parallax calculation unit 102, with an image converted using the above-mentioned planar area estimation result, with an image captured by the other camera, and based on the comparison result, detects that a change has occurred in at least one of the camera parameters: the camera's focal length, optical center, multiple internal parameters of the shear coefficient; multiple external parameters of the camera's up / down, left / right, and front / back directions, as well as the yaw, pitch, and roll directions that are directions around the axes of each of these directions; and the lens distortion parameters.

[0018] When the first parameter change detection unit 104 detects the occurrence of a change in the camera parameter, the output unit 105 outputs information that the camera parameter has changed.

[0019] The image processing device 1 in this embodiment can be used in an automated driving system that is mounted on a vehicle and can recognize obstacles and lanes around the vehicle to allow the vehicle to travel without driver operation, or a driving assistance system that assists the driver in driving the vehicle. The image processing device 1 may have functions for configuring these systems or some of the functions. These functions are not directly related to the present application, so they are not shown here and their description is omitted.

[0020] FIG. 2 is a schematic block diagram showing the hardware configuration of the image processing device.

[0021] The image processing device 1 has a processor 10, a ROM 20, a RAM 21, a camera interface (camera IF) 30, and a network interface (network IF) 31. The processor 10 is a computing device that executes programs to perform various calculations and realizes the various functions provided by the image processing device, in addition to the various units shown in FIG. 1. The ROM is a storage device that stores programs executed by the processor 10 and various constants and setting values ​​required for program execution, and may be a non-volatile storage device such as a flash memory. The RAM 21 is a volatile storage device such as a DRAM (Dynamic Random Access Memory), and is used to store data and various information that the processor temporarily uses when executing a program.

[0022] The camera IF 30 is an interface to which multiple cameras 11 are connected, and captures images sent from the cameras 11 and / or outputs data for controlling the cameras 11. The network IF 31 is connected to a network to which various devices arranged in the vehicle are connected, such as a CAN (Controller Area Network), and is an interface for communicating with other devices. As will be described later, when a change in camera parameters is detected, the network IF 31 sends information indicating this to a navigation system or a communication device that communicates with external devices, and is also used to exchange various information with other devices in the vehicle.

[0023] Note that some or all of the functions of the image processing device 1 may be realized by dedicated hardware such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC) instead of program processing by the processor 10.

[0024] FIG. 3 is a flowchart showing the flow of the detection process in this embodiment.

[0025] The image processing device 1 acquires an image from the camera 11 using the image acquisition unit 101 (S301). The image acquisition unit 101 converts the image acquired in step S301 into an image suitable for subsequent processing. For example, in order to facilitate association of projection positions in multiple images of an object existing in the external world, if the optical axes of the cameras 11-a and 11-b are non-parallel, a converted image (rectified image) is generated by geometrically transforming at least one of the images so that the converted image matches an image that would be captured if the optical axes of the two cameras 11-a and 11-b were parallel. An example of a case in which the optical axes of the cameras are non-parallel is when stereo images are acquired from multiple independent cameras installed on a vehicle under different conditions (S302).

[0026] Next, the image processing device 1 uses the set of multiple converted images generated in process S302, compares one converted image with the other converted image as a reference, calculates the disparity between each image, and obtains disparity information using the disparity calculation unit 102. Hereinafter, the converted image used as the reference is referred to as the reference image, and the other converted image is referred to as the comparison image (S303).

[0027] The image processing device 1 uses the parallax information calculated in step S303 to estimate a planar area of ​​an object existing in the external world by the plane estimation unit 103. For example, the following plane equation

[0028]

number

[0029] A plane having coordinates (x, y, z) expressed by the following equation is estimated (S304).

[0030] After estimating the planar area, the image processing device 1 determines, via the first parameter change detection unit 104, whether or not the planar area estimation process in the plane estimation unit 103 has succeeded in estimating a planar area existing in the external world (S305).

[0031] If the estimation of the planar region is successful, the image processing device 1 uses the information on the planar region estimated in step S304 to geometrically transform the comparison image using the first parameter change detection unit 104 so that the comparison image is an image that assumes the same optical axis as the reference image. That is, the comparison image is geometrically transformed so that the comparison image is an image that would be obtained if it were captured from the same viewpoint as the camera that captured the reference image. For this geometric transformation, for example, a known geometric transformation can be used, which extracts multiple pairs of two-dimensional coordinates x0 of a point on a planar region in the reference image and two-dimensional coordinates x1 of the same point on a planar region in the comparison image, and uses a homography matrix H estimated to satisfy the following equation (S306).

[0032]

number

[0033] Next, the first parameter change detection unit 104 compares the planar regions of the reference image and the transformed image obtained by geometrically transforming the comparison image using brightness information and calculates the match rate. The match rate can be calculated, for example, as the ratio of the number of pixels constituting the planar region in the reference image to the number of pixels constituting the planar region in the transformed comparison image. If the camera parameters of each camera 11 mounted on the vehicle can be estimated without error using a known calibration method and the homography matrix H estimated from the two transformed images in process S205 is also error-free, then theoretically, the planar regions between the two images will perfectly match as images. Therefore, in this case, the match rate is 100%. However, if the camera parameters of at least one of the cameras have changed, the planar regions of the two images will not match, and the match rate will be below 100% (S307).

[0034] The first parameter change detection unit 104 determines whether the calculated match rate is below a predetermined threshold value. The threshold value used here can be determined in consideration of the degree of influence on subsequent processing based on past performance, calculation-based simulations, etc. (S308).

[0035] If the matching rate is equal to or greater than the threshold in step S308, the first parameter change detection unit 104 determines that there is no change in the camera parameters of each camera 11 that would affect subsequent processing, and determines that there is no change in the camera parameters (S309).On the other hand, if the matching rate is less than the threshold, the first parameter change detection unit 104 determines that one of the camera parameters of any of the cameras 11 has changed, which may affect subsequent processing, and determines that there has been a change in one of the camera parameters (S310).

[0036] If plane estimation is not possible in step S306, the first parameter change detection unit 104 determines that a change has occurred in the lens distortion parameters, etc., and determines that a change has occurred in the camera parameters (S312), similar to step S310.

[0037] Finally, the image processing device 1 outputs the determination result in process S209, S210, or S211 via the output unit 105. The determination result output by the image processing device 1 can be used, for example, to be displayed on a display of an in-vehicle device such as a navigation system (not shown), or to be notified to an external device such as a mobile device such as a smartphone owned by a user, or a terminal device owned by a car dealer or a vehicle manufacturer, via wireless or a mobile phone network. Alternatively, the determination result may be used as a trigger to start estimation processing by a known calibration device that is pre-installed in the vehicle and operates while the vehicle is traveling to estimate camera parameters of each camera.

[0038] As described above, according to this embodiment, two camera images are geometrically transformed so that the viewpoints thereof coincide, and changes in camera parameters are detected based on the coincidence rate calculated by comparing the areas projected onto each transformed image, which are recognized as planes existing in the external world. Therefore, it is possible to detect changes in camera parameters such as external parameters in six axial directions related to the position and orientation of the camera, which affect the recognition of the plane area, internal parameters such as focus, optical axis center, and shear, and lens distortion.

[0039] Fig. 4 is a block diagram showing a schematic configuration of an image processing device according to a second embodiment of the present invention. In Fig. 4, components common to those of the first embodiment shown in Fig. 1 are assigned the same reference numerals as in Fig. 1, and overlapping explanations will be omitted below.

[0040] In the image processing device 2 of this embodiment, a planar area specifying unit 201 and a second parameter change detecting unit 202 are provided in parallel with a parallax calculating unit 102, a plane estimating unit 103, and a first parameter change detecting unit 104.

[0041] The planar area identifying unit 201 identifies planar areas existing in the external world that are reflected in the converted image generated from the camera image, using information other than the parallax used for estimation by the plane estimation unit 103. The planar area identifying process performed by the planar area identifying unit 201 can use distance information measured using various distance measuring sensors, such as a LiDAR (Light Detection and Ranging) (not shown). Specifically, this distance information is the distance to structures such as roads and buildings existing in the external world. By comparing this distance information with information in the converted images acquired by each image acquisition unit 101, it is possible to identify planar areas.

[0042] The second parameter change detection unit 202 calculates a match rate by comparing an area in the converted image acquired by the image acquisition unit 101-a, which is identified by the planar area identification unit 201, with a corresponding planar area in the converted image acquired by the image acquisition unit 101-b. If the match rate falls below a predetermined threshold, the second parameter change detection unit 202 determines that a change has occurred in any of the camera parameters of any of the cameras 11.

[0043] When at least one of the first parameter change detection unit 104 and the second parameter change detection unit 202 determines that a change has occurred in the camera parameters, the output unit 203 outputs information indicating that the camera parameters have changed, similar to the output unit 105.

[0044] As described above, according to the image processing device of this embodiment, by adding a plane area identification unit that identifies planes that exist in the external world, when camera parameters have changed, the two units can complement each other's detection results, making it possible to detect more changes in camera parameters.

[0045] The image processing device of the embodiment described above identifies planar areas in each image captured by multiple cameras and detects changes in camera parameters based on the match rate between them. This makes it possible to detect changes in many more camera parameters than before, such as external parameters in six axial directions related to the position and orientation of the camera that affect the recognition of planar areas, internal parameters such as focus, optical axis center, and shear, and lens distortion.

[0046] Furthermore, since detection is performed by determining an area that can be recognized as a plane, for example, the area of ​​the road surface in the image can be used, and changes in camera parameters can be detected even when the vehicle is driving normally.

[0047] Although the present invention has been described above using representative embodiments as examples, the present invention is not limited thereto and can be embodied in various forms without departing from the spirit of the invention as set forth in the claims. For example, the above-described embodiments are provided to facilitate understanding of the present invention, and it is not necessary to include all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another example, or the configuration of one embodiment may be added to the configuration of another embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations. [Explanation of symbols]

[0048] 1···Image processing device, 11-a, 11-b··Camera, 101-a, 101-b··Image acquisition unit, 102···Disparity calculation unit, 103···Plane estimation unit, 104···First parameter change detection unit, 105, 203···Output unit, 201···Planar area identification unit, 202···Second parameter change detection unit

Claims

1. an image acquisition unit that acquires a plurality of images captured by a plurality of cameras with different viewpoints; a parallax calculation unit for calculating parallax information from the plurality of images; a plane estimation unit that estimates a plane area existing in the external world captured in the plurality of images based on the parallax information; a parameter change detection unit that compares a transformed image obtained by transforming one of the plurality of images using the estimation result of the planar region with another image of the plurality of images that is different from the one image, and detects, based on the comparison result, that at least one of camera parameters including a plurality of internal parameters of the camera, a plurality of external parameters, and a lens distortion parameter has changed; an output unit that, when a change in at least one camera parameter is detected by the parameter change detection unit, outputs information that the camera parameter has changed.

2. 2. The image processing device according to claim 1, wherein the converted image is obtained by converting one of the images into an image that would be obtained if the image were captured from the same viewpoint as a camera among the plurality of cameras that captured the other images.

3. The image processing device according to claim 2, wherein the parameter change detection unit obtains a matching rate between the converted image and the other image as a result of the comparison, and detects that at least one of the camera parameters has changed if the matching rate is below a predetermined threshold.

4. 4. The image processing device according to claim 2, wherein the parameter change detection unit detects that at least one of the camera parameters has changed when the plane estimation unit is unable to estimate the planar area.

5. The image processing device according to claim 1 , wherein, when the plurality of cameras form a non-parallel stereo camera, the image acquisition unit converts at least one of the plurality of images to parallelize the plurality of images.

6. The image processing device according to claim 1 further comprises: a planar area specifying unit that specifies a planar area present in the external world captured in the plurality of images by a method different from that of the plane estimation unit; and an additional parameter change detection unit that compares planar areas in the plurality of images identified by the planar area identification unit and detects that at least one of the camera parameters has changed based on a result of the comparison, The image processing device wherein the output unit outputs information that the camera parameter has changed when a change in the camera parameter is detected by at least one of the parameter change detection unit and the other parameter change detection unit.

7. The image processing device according to claim 6 , wherein the planar area specifying unit acquires distance information obtained by a distance measuring sensor that measures distances to structures existing in the external world, and specifies planar areas in the plurality of images based on the distance information.

8. 1. A method for detecting a change in camera parameters in an image processing device for processing images captured by a plurality of cameras connected with different viewpoints, the method comprising: acquiring a plurality of images captured by the plurality of cameras; obtaining parallax information from the plurality of images; estimating a planar area existing in the external world captured in the plurality of images based on the parallax information; comparing a transformed image obtained by transforming one of the plurality of images using the planar region estimation result with another image of the plurality of images that is different from the one image; determining whether any of the camera parameters, including a plurality of intrinsic parameters of the camera, a plurality of extrinsic parameters, and a lens distortion parameter, has changed based on the result of the matching; and When it is determined that any of the camera parameters has changed as a result of the determination, a signal indicating that the camera parameter has changed is output.

9. The method for detecting a change in camera parameters according to claim 8, characterized in that the matching process includes a process of converting one of the images so that it becomes an image that would be obtained if the one of the multiple cameras captured the other image from the same viewpoint as a camera that captured the other image, and acquiring the converted image.

10. the matching process includes a process of obtaining a matching rate between the converted image and the other image, 10. The method for detecting a change in camera parameters according to claim 9, wherein said determining step determines that at least one of said camera parameters has changed if said matching rate is below a predetermined threshold.

11. 11. The method for detecting a change in a camera parameter according to claim 9, wherein the determining step determines that at least one of the camera parameters has changed if the planar region cannot be estimated in the estimating step.

12. 9. The method for detecting a change in a camera parameter according to claim 8, wherein the process of acquiring the images includes a process of converting at least one of the images to parallelize the images when the plurality of cameras constitute a non-parallel stereo camera.

13. The camera parameter change detection method according to claim 8 further comprises: a specification process for specifying a planar area present in the external world captured in the plurality of images by a method different from the estimation; a second determination process of comparing planar areas in the plurality of images identified by the identification process with each other and determining whether or not at least one of the camera parameters has changed based on a result of the comparison; A camera parameter change detection method in which the output process outputs information that the camera parameters have changed when a change in the camera parameters is detected in either the determination process or the second determination process.

14. The method for detecting changes in camera parameters according to claim 13, wherein the identification process acquires distance information obtained by a ranging sensor that measures the distance to a structure existing in the external world, and identifies planar areas within the plurality of images based on the distance information.

Images