On-vehicle image processing device and image processing method for on-vehicle equipment

Abstract

The present invention comprises: a first image correction unit that uses an image captured by a first camera, which is set beforehand from among a plurality of imaging units mounted at the periphery of a vehicle, as a reference to correct the image deviation of an image captured by a second camera having a resolution lower than or equal to the resolution of the first camera; and a first stereo recognition processor that performs stereo recognition processing using the image captured by the first camera and the image captured by the second camera and corrected by the first image correction unit. The present invention further comprises a second stereo recognition processor that performs stereo recognition processing using the image captured by the second camera and an image obtained by correcting an image captured by a third camera.

Description

In-vehicle image processing device and image processing method for in-vehicle equipment

[0001] The present invention relates to an in-vehicle image processing device and an image processing method for in-vehicle equipment.

[0002] In order to enable autonomous driving and driving assistance in automobiles and other vehicles, development is underway to develop sensing technologies that use on-board cameras to capture images of the surrounding area and detect objects and people around the vehicle.

[0003] One method for recognizing objects from images taken with a camera is stereo processing, in which two cameras capture images of the same object. Stereo processing makes it possible to measure the distance to the object captured from images taken with the two cameras. To measure the distance, the positions and angles at which the two cameras capture images must be adjusted correctly, and the two cameras must be photographed without any axis misalignment. If these adjustments are not made correctly, the matching accuracy of the two images, or in other words, distance measurement performance, will decrease.

[0004] Patent Document 1 describes a technology for an in-vehicle stereo camera device that uses a stereo camera, in which stereo processing is performed using only the central part of the image for distant objects and the wide-angle area of ​​the periphery for nearby objects, thereby ensuring matching accuracy for distant objects in telephoto mode while achieving low costs.

[0005] Japanese Patent Application Laid-Open No. 2019-178871

[0006] Meanwhile, the number of on-board cameras installed in a single vehicle is on the rise. That is, with the advancement of autonomous driving functions and driving assistance functions, there is a trend toward installing a large number of cameras, such as a front camera that captures images in front of the vehicle, a rear camera that captures images behind the vehicle, and side cameras that capture images to the sides. In this case, if each camera were to be a stereo camera, twice the number of cameras would be required, which would undesirably complicate the configuration and increase costs.

[0007] For this reason, a method has been considered for performing distance measurement by stereo processing using only the overlapping areas of images captured by multiple cameras with different angles of view. With this configuration, distance measurement can be performed using stereo processing without substantially increasing the number of on-board cameras.

[0008] When performing stereo processing with such a configuration, unless the combination of multiple cameras for stereo processing is appropriately selected, the range that can be measured by the stereo processing becomes limited. Specific details of this problem will be described in the embodiment described later. However, the problem that arises when performing appropriate stereo processing from images captured with different angles of view cannot be solved by conventional stereo processing in which two cameras capture approximately the same range. Patent Document 1 describes performing stereo processing from a limited area, such as the central portion of the image, but this method targets only two cameras and is therefore not applicable when more cameras are installed on a vehicle and multiple sets of stereo processing are performed.

[0009] An object of the present invention is to provide an in-vehicle image processing device and an image processing method for in-vehicle equipment that can perform highly accurate distance measurement using a plurality of in-vehicle cameras.

[0010] In order to solve the above problem, for example, the configuration described in the claims is adopted. The present application includes multiple means for solving the above problem, and one example thereof is an in-vehicle image processing device including an image acquisition unit that acquires images captured by multiple cameras mounted around the vehicle, a first image correction unit that corrects image shifts of images captured by a second camera of the multiple cameras, the second camera having a lower or equal resolution than the first camera, based on an image captured by a preset first camera of the multiple cameras, a first stereo recognition processing unit that performs stereo recognition processing using the images captured by the first camera and the images captured by the second camera corrected by the first image correction unit, a second image correction unit that corrects image shifts of images captured by a third camera having a lower or equal resolution than the second camera, based on an image captured by the second camera, and a second stereo recognition processing unit that performs stereo recognition processing using the images captured by the second camera and the images captured by the third camera corrected by the second image correction unit.

[0011] According to the present invention, since the reference image is appropriately determined when performing multiple sets of stereo recognition processing, each stereo processing can be performed without increasing the cumulative error or the burden of the stereo recognition processing. Therefore, distance measurement processing of objects around the vehicle can be performed with high accuracy. Problems, configurations, and effects other than those described above will become clear from the description of the following embodiments.

[0012] FIG. 1 is a configuration diagram showing an example of an in-vehicle image processing device according to an embodiment of the present invention. FIG. 2 is a configuration diagram showing an example of an image correction unit of the in-vehicle image processing device according to an embodiment of the present invention. FIG. 3 is a configuration diagram showing an example of an image selection unit of the in-vehicle image processing device according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of an imaging range of an image to be processed by the in-vehicle image processing device according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of a stereo combination range of each image when stereo processing is performed by the in-vehicle image processing device according to an embodiment of the present invention. FIG. 6 is a diagram showing an example of image combination (camera combination) for stereo processing by the in-vehicle image processing device according to an embodiment of the present invention. FIG. 7 is a diagram showing an example of a reference image and a corrected image during stereo processing by the in-vehicle image processing device according to an embodiment of the present invention. FIG. 8 is a diagram showing an example of the principle of image correction by the in-vehicle image processing device according to an embodiment of the present invention. FIG. 9 is a diagram showing a conventional example when multiple stereos are performed.

[0013] An embodiment of the present invention will now be described with reference to the accompanying drawings. [Configuration of an In-Vehicle Image Processing Device] FIG. 1 shows an example of the overall configuration of an in-vehicle image processing device according to this embodiment. The in-vehicle image processing device 20 according to this embodiment is mounted on a vehicle such as an automobile and processes images from seven cameras 11-17, which are multiple imaging units installed on the vehicle. The installation status of the seven cameras 11-17 on the vehicle will be described later. Each of the cameras 11-17 is a video camera that captures images at a constant frame rate, and at least a portion of the range captured by each camera 11-17 overlaps with the range captured by another camera among the cameras 11-17. Seven cameras (imaging units) 11-17 is just an example, and other numbers may be used.

[0014] The in-vehicle image processing device 20 includes an image acquisition unit 21, an image storage unit 22, an object recognition processing unit 23, an image adjustment unit 24, an image correction unit 25, an image selection unit 26, and a stereo recognition processing unit 27. The image acquisition unit 21 performs image acquisition processing of image signals captured by the seven cameras 11 to 17. The image signals of each camera acquired by the image acquisition unit 21 are supplied to the image storage unit 22. The image storage unit 22 temporarily stores the images captured by the seven cameras 11 to 17 that are supplied to it.

[0015] The object recognition processing unit 23 performs a process of recognizing objects (objects) around the vehicle from the images captured by each of the cameras 11 to 17, and outputs information about the recognized objects. When performing this recognition process, information about the distance of the recognized object from the vehicle body is added based on information obtained by the stereo recognition processing in the stereo recognition processing unit 27. Speed ​​may also be recognized from changes in distance. The image adjustment unit 24 performs an adjustment process to make the images captured by each of the cameras 11 to 17 into appropriate images. This adjustment process includes various image processes such as adjusting the size and correcting the brightness of the images to properly perform selection by the image selection unit 26 and recognition by the stereo recognition processing unit 27. One of the adjustments performed by the image adjustment unit 24 is a process of detecting the amount of deviation between each captured image.

[0016] The image correction unit 25 performs a process of correcting misalignment of each image based on the misalignment detection by the image adjustment unit 24. The process of correcting misalignment of images includes, for example, correction of pitching, rolling, and yaw misalignment, and more specifically, vertical offset correction, horizontal offset correction, and angle correction. In this embodiment, the stereo recognition processing unit 27 performs a plurality of sets of recognition processing, so that a configuration is adopted in which correction of a plurality of images is performed simultaneously.

[0017] 2, the image correction unit 25 is provided with a plurality of processing units, namely, a first image correction unit 25a, a second image correction unit 25b, etc., and each of the image correction units 25a, 25b, etc. performs a different image correction (first image correction processing, second image correction processing, third image correction processing, etc.). A specific example of the correction processing performed by the image correction unit 25 will be described later in the description of FIG. 8.

[0018] The image selection unit 26 performs image selection processing for stereo recognition using the images corrected by the image correction unit 25 and the images stored in the image storage unit 22. In this embodiment, the stereo recognition processing unit 27 performs multiple sets of recognition processing, so the configuration is such that multiple images are selected simultaneously. In other words, the image selection unit 26 also has processing units that perform multiple sets of selection, and each image selection unit selects a different image. A specific example of image selection by the image selection unit 26 will be described later in the description of FIG. 5.

[0019] The stereo recognition processing unit 27 performs a recognition process for an object in the two images selected by the image selection unit 26, and performs a ranging process for recognizing the distance from the vehicle to the object based on the parallax between the two images. That is, as shown in Fig. 3, the stereo recognition processing unit 27 also has a first stereo recognition processing unit 27a, a second stereo recognition processing unit 27b, a third stereo recognition processing unit 27c, a fourth stereo recognition processing unit 27d, etc., and is configured to perform multiple sets of stereo recognition processes simultaneously. However, Fig. 3 shows only the first stereo recognition processing unit 27a and the second stereo recognition processing unit 27b, and does not show the third stereo recognition processing unit and the fourth stereo recognition processing unit.

[0020] Note that "stereo recognition processing performed simultaneously" here means that processing is performed almost simultaneously. In addition to the configuration in which processing is performed in parallel by multiple processing units as shown in Figures 2 and 3, each set of processing may be performed sequentially in a time-division manner by a limited number of processing units (e.g., one stereo recognition processing unit) within one frame period in which an image is supplied from the camera. Therefore, the multiple image correction units and multiple stereo recognition processing units shown in Figures 2 and 3 exist in multiple numbers from the perspective of the flow of image signals to be processed, but there is no guarantee that there will be multiple units in the hardware configuration. Distance information obtained by the ranging processing is supplied to the object recognition processing unit 23.

[0021] The in-vehicle image processing device 20 shown in Fig. 1 can be configured, for example, by a computer, in which a CPU (Central Processing Unit) 20a, a memory 20b, an image input unit 20c, an interface 20d, and an image output unit 20e are connected by a bus line, as shown in the lower part of Fig. 1.

[0022] The CPU 20a is a processor that executes a program stored in the memory 20b, thereby configuring each processing unit in the memory 20b and executing image recognition processing. The processing units configured in the memory 20b under the control of the CPU 20a are the object recognition processing unit 23, image adjustment unit 24, image correction unit 25, image selection unit 26, and stereo recognition processing unit 27 described above.

[0023] The memory 20b is a storage unit that serves as a work area for executing arithmetic processing under the control of the CPU 20a, and stores programs and various data. The image input unit 20c processes the input of images captured by the cameras 11 to 17. The interface 20d exchanges information with other control devices (not shown) in the vehicle. For example, it transmits information about objects and distances around the vehicle obtained by the in-vehicle image processing device 20 to the other control devices. The image output unit 20e outputs images processed by the in-vehicle image processing device 20 to be displayed on a display unit (not shown) in the vehicle.

[0024] Note that the in-vehicle image processing device 20 configured to perform arithmetic processing based on a program installed therein is merely an example, and other configurations are also possible. For example, part or all of the in-vehicle image processing device 20 may be realized by dedicated hardware such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0025] [Example of angle of view of cameras mounted on a vehicle] Figure 4 shows an example of the angle of view of the seven cameras 11 to 17 that supply captured images to the in-vehicle image processing device 20 of this embodiment. In the vehicle shown in the center of Figure 4, the right side is the front and the left side is the rear. The upper side of Figure 4 is the left side, and the lower side of Figure 4 is the right side. The angle of view of the seven cameras 11 to 17 will be described below.

[0026] The first camera 11 is a front narrow-angle camera that captures images within a front narrow angle C1 of the vehicle. Here, the front narrow angle is, for example, 30°. The first camera 11, which captures images within this front narrow angle C1, has the largest number of pixels among the seven cameras 11 to 17 and captures images with high resolution. The second camera 12 is a front wide-angle camera that captures images within a front wide angle C2 of the vehicle. Here, the front wide angle is, for example, 120°. The second camera 12, which captures images within the front wide angle C2, has the same number of pixels as the first camera 11 or fewer pixels than the first camera 11 and captures images with the same or lower resolution than the first camera 11.

[0027] The third camera 13 is a side front camera that captures an image in the range of the side front C3 on the left side of the vehicle. The side front here is, for example, 120°. The fourth camera 14 is a side front camera that captures an image in the range of the side front C4 on the right side of the vehicle. The side front here is, for example, 120°. The third camera 13 and the fourth camera 14 capture images with the same number of pixels as the first camera 11 and the second camera 12, or fewer pixels than the first camera 11 and the second camera 12, and with the same resolution as the first camera 11, or lower than the first camera 11. In addition,

[0028] The fifth camera 15 is a side rear camera that captures an image of a range C5 of the left side of the vehicle. Here, the side rear is, for example, 120°. The sixth camera 16 captures an image of a range C6 of the right side of the vehicle. Here, the side rear is, for example, 120°. The fifth camera 15 and the sixth camera 16 capture images having the same number of pixels as the first camera 11 and the second camera 12, or fewer pixels than the first camera 11 and the second camera 12, and the same or lower resolution than the first camera 11. Furthermore, when compared with the third camera 13 and the fourth camera 14, the fifth camera 15 and the sixth camera 16 capture images having the same number of pixels as the third camera 13 and the fourth camera 14, or fewer pixels than the third camera 13 and the fourth camera 14, and the same or lower resolution than the third camera 13 and the fourth camera 14. Furthermore, when compared with the seventh camera 17 (rear camera) described below, the fifth camera 15 and the sixth camera 16 capture images with the same or fewer pixels as the seventh camera 17 and with the same or lower resolution than the seventh camera 17. The seventh camera 17 is a rear camera that captures images of the range C7 of the rear of the vehicle. Here, the rear is, for example, 30°.

[0029] In this manner, in this embodiment, seven cameras 11 to 17 are used to capture images of the entire 360° horizontal periphery of the vehicle. As shown in Fig. 4, the imaging ranges C1 to C7 of the cameras 11 to 17 at least partially overlap with the imaging ranges of the other cameras. In this embodiment, stereo processing is performed using the overlapping areas of the imaging ranges.

[0030] 5 shows an example of images to be subjected to stereo processing. An image P1 captured at a front narrow angle C1 by the first camera 11 has an image range that entirely overlaps with an image P2 captured at a front wide angle C2 by the second camera 12. The in-vehicle image processing device 20 performs one set of stereo processing (first stereo recognition processing) using almost the entire range P1a of the image P1 captured at the front narrow angle C1 and a substantially central range P2a of the image P2 captured at the front wide angle C2.

[0031] The right half P2b of the image P2 captured by the second camera 12 at the front wide angle C2 overlaps with the left half P4a of the image P4 of the side front C4 captured by the fourth camera 14. The in-vehicle image processing device 20 performs one set of stereo processing (second stereo recognition processing) using the right half P2b of the front wide angle C2 and the left half P4a of the right side front C4.

[0032] Furthermore, the left half P2c of the image P2 captured by the second camera 12 at the front wide angle C2 overlaps with the right half P3a of the image P3 of the side front C3 captured by the third camera 13. The in-vehicle image processing device 20 performs one set of stereo processing (third stereo recognition processing) using the left half P2c of the front wide angle C2 and the right half P3a of the left side front C3.

[0033] Furthermore, the rightmost range P4b of the image P4 of the right side front C4 captured by the fourth camera 14 overlaps with the leftmost range P6a of the image P6 of the side rear C6 captured by the sixth camera 16. The in-vehicle image processing device 20 performs one set of stereo processing (fourth stereo recognition processing) using the rightmost range P4b of the right side front C4 and the leftmost range P6a of the right side rear C6.

[0034] Furthermore, the left edge range P3b of the image P3 of the left side front C3 captured by the third camera 13 overlaps with the right edge P5a of the image P5 of the left side rear C5 captured by the fifth camera 15. The in-vehicle image processing device 20 performs one set of stereo processing (fifth stereo recognition processing) using the left edge range P3b of the left side front C3 and the right edge P5a of the left side rear C5.

[0035] Furthermore, a portion P6b of the right edge of the image P6 of the side rear C6 captured by the sixth camera 16 overlaps in its imaging range with a left half P7b of the image P7 of the rear C7 captured by the seventh camera 17. The in-vehicle image processing device 20 performs one set of stereo processing (sixth stereo recognition processing) using the portion P6b of the right edge of the image P6 of the side rear C6 and the left half P7b of the image P7 of the rear C7.

[0036] Furthermore, a portion P5b of the left edge of the image P5 of the side rear C5 captured by the fifth camera 15 overlaps with a right half P7a of the image P7 of the rear C7 captured by the seventh camera 17. The in-vehicle image processing device 20 performs one set of stereo processing (seventh stereo recognition processing) using the portion P5b of the right edge of the image P5 of the side rear C5 and the right half P7a of the image P7 of the rear C7.

[0037] Although not shown in Figure 5, since the image P1 of the front narrow angle C1 and the images P3 and P4 of the side front C3 and C4 also have some overlapping imaging ranges, stereo processing may be performed using these images.

[0038] The in-vehicle image processing device 20 performs these stereo processes simultaneously. However, "simultaneously" here means that they are performed almost simultaneously, and they may be performed sequentially in a time-division manner, for example. When performing each stereo process, in order to align the two images, it is necessary to correct the image from one camera as a reference image and the image from the other camera so that the two images can be properly processed in stereo.

[0039] 6 shows an example of a combination of a reference image and an image to be corrected when the in-vehicle image processing device 20 performs image combination (camera combination) for stereo processing. When performing stereo processing (first stereo recognition processing) using an image P1 with a front narrow angle C1 and an image P2 with a front wide angle C2, the image P1 with the front narrow angle C1 captured by the first camera 11 is used as the reference. Then, the image P2 with the front wide angle C2 captured by the second camera 12 is used as the image to be corrected for stereo processing.

[0040] When performing stereo processing (processing other than the first to seventh stereo recognition processes) using the image P1 of the narrow front angle C1 and the images P3 and P4 of the side fronts C3 and C4, the image P1 of the narrow front angle C1 captured by the first camera 11 is used as a reference. The images P3 and P4 of the side fronts C3 and C4 captured by the third camera 13 and the fourth camera 14 are used as correction target images for the stereo processing. However, the stereo processing that combines the image P1 of the narrow front angle C1 with the images P3 and P4 of the side fronts C3 and C4 does not necessarily have to be performed.

[0041] When performing stereo processing (second stereo recognition processing and third stereo recognition processing) using the image P2 of the front wide-angle C2 and the images P3 and P4 of the side front C3 and C4, the image P2 of the front wide-angle C2 captured by the second camera 12 is used as a reference, and the images P3 and P4 of the side front C3 and C4 captured by the third camera 13 and the fourth camera 14 are used as correction target images for the stereo processing.

[0042] When performing stereo processing (sixth stereo recognition processing and seventh stereo recognition processing) using the image P7 of the rear C7 and the images P5 and P6 of the side rear C5 and C6, the image P7 of the rear C7 captured by the seventh camera 17 is used as a reference. The images P5 and P6 of the side rear C5 and C6 captured by the fifth camera 15 and the sixth camera 16 are used as correction target images for the stereo processing.

[0043] When performing stereo processing (fourth stereo recognition processing and fifth stereo recognition processing) using the images P3 and P4 of the side fronts C3 and C4 and the images P5 and P6 of the side rears C5 and C6, the images P3 and P4 of the side fronts C3 and C4 captured by the fifth camera 15 and the sixth camera 16 are used as references. The images P5 and P6 of the side rears C5 and C6 captured by the fifth camera 15 and the sixth camera 16 are then used as correction target images for the stereo processing.

[0044] 7 shows a process for obtaining a reference image and a corrected image from captured images when performing a first stereo recognition process using an image P1 with a narrow front angle C1 and an image P2 with a wide front angle C2. In the first stereo recognition process, the image P1 with the narrow front angle C1 serves as the reference, so the captured image P1 becomes the image for stereo processing as is, and almost the entire range of the captured image P1 becomes one of the images P1a used in the stereo processing.

[0045] Since image P2 of the front wide-angle lens C2 is an image to be corrected in the first stereo recognition process, offset correction and angle correction are performed so that the range P2a of the extracted image becomes the same as the range of image P1a. Correction processing is also performed so that the image can be treated as an image of the same size as the other image P1a. When this size adjustment is performed, the number of pixels in the range P2a of image P2 may be increased by interpolation or the like.

[0046] FIG. 8 shows specific examples of offset correction and angle correction by the image correction unit 25. For example, as shown in the upper part of FIG. 8 , assume that two images PL1 and PR1 undergoing stereo processing have an offset in the vertical direction by a predetermined amount ΔY. In this case, the reference image (image PL1 in the example of FIG. 8 ) is not corrected, and roll correction is performed to shift the image PR1 to be corrected by a predetermined amount ΔY in the vertical direction. Also, although not shown, when there is a misalignment between the two images in the horizontal direction, the reference image PL1 is not corrected, and pitch correction is performed to shift the image PR1 to be corrected by a predetermined amount ΔX in the horizontal direction. Also, as shown in the lower part of FIG. 8 , assume that two images PL2 and PR2 undergoing stereo processing have an angular misalignment by a predetermined amount ΔR. In this case, the reference image (image PL2 in the example of FIG. 8 ) is not corrected, and yaw misalignment correction is performed to rotate the image PR2 to be corrected by a predetermined amount ΔR.

[0047] [Effects of Processing in the Present Embodiment] As described above, when performing stereo processing using multiple cameras mounted on a vehicle, errors do not accumulate by determining the reference image and the image to be corrected as described in FIG. 6 . That is, in the first stereo recognition processing, object recognition and distance measurement can be performed through good stereo processing by using the image captured by the first camera 11 (front narrow-angle camera) with high resolution as the reference. In the second stereo recognition processing and the third stereo recognition processing, the image captured by the second camera 12 is used as the reference, and therefore good stereo processing can be performed. In addition, in the fourth stereo recognition processing and the fifth stereo recognition processing, the image captured by the third camera 13 or the fourth camera 14 is used as the reference, and therefore good stereo processing can be performed.

[0048] On the other hand, the sixth stereo recognition process and the seventh stereo recognition process use the image captured by the high-resolution seventh camera 17 (rear camera) as a reference, which also enables good object recognition and distance measurement through stereo processing. Furthermore, in each stereo process, the image captured by the camera is used as the reference image as is for one image, and only the other image is corrected, which reduces the number of images to be temporarily stored in the image processing device 20 for correction, and reduces the burden on the image processing device 20 during image processing.

[0049] [Comparative Example When the Processing of the Present Embodiment is Not Performed] Here, in order to demonstrate the effect of preventing errors from accumulating by performing the stereo processing of the present embodiment, processing when performing conventional stereo processing of multiple sets is shown in FIG. 9. The example of FIG. 9 is an example in which the reference image when performing the stereo processing is one of the images in turn. That is, when performing the first stereo recognition processing using an image with a front narrow angle C1 and an image with a front wide angle C2, a pitching ΔX is applied to the image with the front wide angle C2. １ , roll ΔY １ , yaw Δθ １ Furthermore, when the third stereo recognition process is performed using the image of the front wide angle C2 and the image of the side front C4, the image of the side front C4 is corrected by applying a pitching ΔX ２ , roll ΔY ２ , yaw Δθ ２ Correction is performed.

[0050] When the fifth stereo recognition process is performed using the image of the side front C4 and the image of the side rear C6, the image of the side front C4 corrected in the third stereo recognition process is used as a reference, and a pitching ΔX is applied to the image of the side rear C6. ３ , roll ΔY ３ , yaw Δθ ３ Correction is performed.

[0051] In this way, correction is performed sequentially on the images of the surroundings of the vehicle at adjacent angles of view, and correction is then performed sequentially on the images of the surroundings of the vehicle. When the process goes through one cycle and the second stereo recognition process is performed on the image of the front wide angle C2 and the image of the side front C3, the image of the side front C3 is used as a reference and the image of the front wide angle C2 is subjected to a pitching ΔX ７ , roll ΔY ７ , yaw Δθ ７ The correction will be performed.

[0052] During this second stereo recognition process, the corrected image of the side front C3, which is an accumulated correction of each previous stereo process, is used as a reference, and each correction ΔX is applied to the image of the front wide angle C2. ７ , ΔY ７ , Δθ ７ Therefore, the range becomes narrower than the range in which the image of the front wide angle C2 is used during the third stereo recognition process. In other words, in order to perform multiple stereo processes, corrections are made cumulatively, and as a result, images with low resolution are used, and the number of pixels of the image sensor of the camera cannot be effectively used. In contrast, in the case of this embodiment, image corrections are not made cumulatively, and stereo processing that effectively uses the number of pixels of the image sensor of the camera can be performed.

[0053] [Modifications] The embodiments described above have been described in detail to facilitate understanding of the present invention, and are not necessarily limited to those including all of the configurations described. For example, the range captured by the seven cameras shown in Fig. 5 is one example, and the number of cameras and the respective capturing angles of view may be different.

[0054] 5, seven cameras are used to capture images of the entire area around the vehicle without any blind spots. However, some cameras may be omitted, and some areas around the vehicle may not be captured. In this case, for example, a stereo processing system for a first camera group based on the front camera (front narrow-angle camera or front wide-angle camera) and a stereo processing system for a second camera group based on the rear camera may be prepared. In other words, by prioritizing the images captured by the front camera or rear camera, which have relatively high resolution, as the reference image and correcting the images captured by the side cameras, stereo processing that effectively utilizes the resolution of the cameras can be performed, similar to the embodiment described above.

[0055] 1 shows only the control lines and information lines that are considered necessary for explanation, and does not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0056] 1 is an example in which each processing unit is configured on a computer by executing a program. In this case, the program may be stored in a recording medium such as an external memory, an IC card, an SD card, or an optical disk, and transferred to the computer (the in-vehicle image processing device 20).

[0057] 11...First camera (front narrow-angle camera), 12...Second camera (front narrow-angle camera), 13...Third camera (front side camera), 14...Fourth camera (front side camera), 15...Fifth camera (side rear camera), 16...Sixth camera (side rear camera), 17...Seventh camera (rear camera) 20...In-vehicle image processing device, 20a...CPU, 20b...Memory, 20c...Image input unit, 20d...Interface, 20e...Image output unit, 21...Image acquisition unit, 22...Image storage unit, 23...Object recognition processing unit, 24...Image adjustment unit, 25...Image correction unit, 26...Image selection unit, 26b...Second image selection unit, 27...Stereo recognition processing unit

Claims

1. An in-vehicle image processing device comprising: an image acquisition unit that acquires images captured by multiple cameras mounted around a vehicle; a first image correction unit that corrects image shift of an image captured by a second camera of the multiple cameras, the second camera having a lower resolution than or equal to that of the first camera, based on an image captured by a predetermined first camera of the multiple cameras; a first stereo recognition processing unit that performs stereo recognition processing using the image captured by the first camera and the image captured by the second camera corrected by the first image correction unit; a second image correction unit that corrects image shift of an image captured by a third camera, the second camera having a lower resolution than or equal to that of the second camera, based on the image captured by the second camera; and a second stereo recognition processing unit that performs stereo recognition processing using the image captured by the second camera and the image captured by the third camera corrected by the second image correction unit.

2. The vehicle-mounted image processing device described in claim 1, wherein the first camera is a front narrow-angle camera mounted at the front of the vehicle and having a first field of view, the second camera is a front wide-angle camera mounted at the front of the vehicle, whose imaging field of view partially overlaps with that of the front narrow-angle camera and has a second field of view wider than the first field of view, and the third camera is a front side camera mounted on the front side of the vehicle and whose imaging field of view partially overlaps with that of the front wide-angle camera.

3. The in-vehicle image processing device according to claim 1, further comprising: a third image correction unit that corrects image shift of an image captured by a fourth camera having a lower or equal resolution than the third camera based on an image captured by the third camera; and a third stereo recognition processing unit that performs stereo recognition processing using the image captured by the third camera and the image captured by the fourth camera corrected by the third image correction unit.

4. The vehicle-mounted image processing device described in claim 3, wherein the first camera is a front narrow-angle camera mounted at the front of the vehicle and having a first field of view; the second camera is a front wide-angle camera mounted at the front of the vehicle, whose imaging field of view partially overlaps with that of the front narrow-angle camera, and whose second field of view is wider than the first field of view; the third camera is a front side camera mounted at the front side of the vehicle and whose imaging field of view partially overlaps with that of the front wide-angle camera; and the fourth camera is a side rear camera mounted at the rear side of the vehicle and whose imaging field of view partially overlaps with that of the front side camera.

5. The in-vehicle image processing device according to claim 4, further comprising: a rear camera mounted at the rear of the vehicle; a fourth image correction unit that corrects image shift of an image captured by the fourth camera having a lower or equal resolution than the rear camera based on an image captured by the rear camera; and a fourth stereo recognition processing unit that performs stereo recognition processing using the image captured by the rear camera and the image captured by the fourth camera corrected by the fourth image correction unit.

6. The in-vehicle image processing device according to claim 1, wherein the first camera, the second camera and the third camera constitute a first camera group that captures an image of the range from the front to the sides of the vehicle, and further comprising a fourth camera and a fifth camera that constitute a second camera group that captures an image of the range from the rear to the sides of the vehicle, and wherein the images captured by the fourth camera and the fifth camera that constitute the second camera group are subjected to stereo recognition processing based on the image captured by the fourth camera or the fifth camera.

7. The in-vehicle image processing device according to claim 1, further comprising a recognition processing unit that performs distance measurement processing of objects around the vehicle from two images that have been subjected to stereo recognition processing by the first stereo recognition processing unit and the second stereo recognition processing unit.

8. An image processing method for an in-vehicle device, comprising: an image acquisition process for acquiring images captured by a plurality of cameras mounted around a vehicle; a first image correction process for correcting image shift of an image captured by a second camera of the plurality of cameras having a lower resolution than or equal to that of the first camera, based on an image captured by a predetermined first camera of the plurality of cameras; a first stereo recognition process for performing stereo recognition processing using the image captured by the first camera and the image captured by the second camera corrected by the first image correction process; a second image correction process for correcting image shift of an image captured by a third camera having a lower resolution than or equal to that of the second camera, based on the image captured by the second camera; and a second stereo recognition process for performing stereo recognition processing using the image captured by the second camera and the image captured by the third camera corrected by the second image correction process.