Automotive image processing device and automotive image processing method

Abstract

The present invention comprises: a first image correction unit that uses an image captured by a first camera, which is set beforehand from among a plurality of imaging units mounted at the periphery of a vehicle, as a reference to correct the image deviation of an image captured by a second camera having a resolution lower than or equal to the resolution of the first camera; and a first stereo recognition processor that performs stereo recognition processing using the image captured by the first camera and the image captured by the second camera and corrected by the first image correction unit. The present invention further comprises a second stereo recognition processor that performs stereo recognition processing using the image captured by the second camera and an image obtained by correcting an image captured by a third camera.

Description

[Technical Field]

[0001] The present invention relates to an on-vehicle image processing apparatus and an image processing method for on-vehicle equipment. [Background Art]

[0002] For vehicles such as automobiles, development of sensing technology that photographs the surroundings with a camera mounted on the vehicle and detects objects or people around the own vehicle for automatic driving or driving assistance is progressing.

[0003] As one of methods for recognizing an object from images captured by cameras, there is stereo processing in which two cameras capture the same object. By performing stereo processing, distance measurement that detects the distance to the captured object from the images captured by the two cameras becomes possible. For distance measurement, it is necessary to correctly adjust the positions and capturing angles of the two cameras so that capturing is performed without axis misalignment between the two cameras. If the adjustment is not correct, this leads to a decrease in the matching accuracy of the two images, that is, distance measurement performance.

[0004] Patent Document 1 describes a technology for an on-vehicle stereo camera device using a stereo camera, which executes stereo processing by using only a central portion of an image for distant objects and using a wide-angle area in a peripheral portion for nearby objects, thereby achieving cost reduction while ensuring matching accuracy for distant objects in telephoto. [Prior Art Documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Unexamined Patent Publication No. 2019-178871 [Summary of the Invention] [Problem to be Solved by the Invention]

[0006] Incidentally, the number of in-vehicle cameras mounted on a single vehicle tends to increase. That is, with the advancement of autonomous driving and driver assistance functions, there is a trend to install multiple cameras, such as side cameras to capture the sides, in addition to the front camera that captures the area in front of the vehicle and the rear camera that captures the area behind the vehicle. However, if each of these cameras were to be a stereo camera, twice the number of cameras would be required, which would complicate the configuration and increase costs, making it undesirable.

[0007] Therefore, it is being considered to perform stereo processing on only the overlapping areas of images captured by multiple cameras, each with a different field of view, to perform distance measurement. With this configuration, distance measurement can be performed using stereo processing without effectively increasing the number of in-vehicle cameras.

[0008] When performing stereo processing with this configuration, if the combination of multiple cameras for stereo processing is not properly selected, there is a problem in that the range that can be measured by stereo processing becomes limited. The specific details of this problem will be explained in the embodiment examples described later, but the problem that arises from performing appropriate stereo processing on images with different fields of view cannot be solved with conventional stereo processing where two cameras capture approximately the same area. Although Patent Document 1 describes a method of performing stereo processing from a specific area, such as the central part of the image, it only applies to two cameras and cannot be used when multiple cameras are mounted on a vehicle and multiple sets of stereo processing are required.

[0009] The present invention aims to provide an in-vehicle image processing device and an image processing method for in-vehicle equipment that can perform highly accurate distance measurement using multiple in-vehicle cameras. [Means for solving the problem]

[0010] To solve the above problems, for example, the configuration described in the claims is adopted. The present invention includes several means for solving the above problem, but one example is an in-vehicle image processing device that arranges multiple cameras to photograph the entire circumference of a vehicle, and overlaps the shooting range of each camera with that of an adjacent camera, and performs stereo recognition at the point where the shooting ranges overlap. The system includes multiple cameras, at least a front narrow-angle camera that captures a narrow angle of the front of the vehicle, a front wide-angle camera that captures a wide angle of the front of the vehicle, a left front side camera that captures the left front side of the vehicle, and a right front side camera that captures the right front side of the vehicle. A first image correction unit performs image shift correction of the front wide-angle camera based on the image of the front narrow-angle camera, A first stereo recognition processing unit performs stereo recognition processing using an image captured by a front narrow-angle camera and an image captured by a front wide-angle camera that has been corrected by a first image correction unit. A second image correction unit corrects the image misalignment of the left front side camera based on the image from the front wide-angle camera, A second stereo recognition processing unit performs stereo recognition processing using an image captured by the front wide-angle camera and an image captured by the left front side camera that has been corrected by the second image correction unit. Front wide-angle camera Image A third image correction unit performs image misalignment correction of the right front side camera based on this, The system includes a third stereo recognition processing unit that performs stereo recognition processing using an image captured by a front wide-angle camera and an image captured by a right front-side camera that has been corrected by a third image correction unit. [Effects of the Invention]

[0011] According to the present invention, since the reference image used when performing multiple sets of stereo recognition processing is appropriately determined and the processing is carried out accordingly, the cumulative error is not increased, and each stereo processing can be performed without increasing the burden of stereo recognition processing. Therefore, distance measurement processing of objects around the vehicle can be performed with high accuracy. Problems, configurations and effects other than those described above will be clarified by the following description of embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] [Figure 1] It is a configuration diagram showing an example of an in-vehicle image processing device according to an embodiment of the present invention. [Figure 2] It is a configuration diagram showing an example of an image correction unit of an in-vehicle image processing device according to an embodiment of the present invention. [Figure 3] It is a configuration diagram showing an example of an image selection unit of an in-vehicle image processing device according to an embodiment of the present invention. [Figure 4] It is a diagram showing an example of an imaging range of an image processed by an in-vehicle image processing device according to an embodiment of the present invention. [Figure 5] It is a diagram showing an example of a stereo combination range of each images when performing stereo processing by an in-vehicle image processing device according to an embodiment of the present invention. [Figure 6] It is a diagram showing an example of image combination (camera combination) in stereo processing by an in-vehicle image processing device according to an embodiment of the present invention. [Figure 7] It is a diagram showing an example of a reference image and a corrected image during stereo processing by an in-vehicle image processing device according to an embodiment of the present invention. [Figure 8] It is a diagram showing an example of the principle of image correction by an in-vehicle image processing device according to an embodiment of the present invention. [Figure 9] It is a diagram showing a conventional example when a plurality of stereo processes are performed. MODE FOR CARRYING OUT THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. [Configuration of In-vehicle Image Processing Device] Fig. 1 shows an example of the overall configuration of the in-vehicle image processing device according to the present embodiment. The in-vehicle image processing device 20 of the present embodiment is mounted on a vehicle such as an automobile, and processes images captured by seven cameras 11 to 17, which are a plurality of imaging units installed on the vehicle. The installation status of the seven cameras 11 to 17 on the vehicle will be described later. Each of the cameras 11 to 17 is a video camera that captures images at a constant frame rate, and at least a part of the imaging range of each camera 11 to 17 overlaps with the imaging range of another camera among the cameras 11 to 17. The number of cameras (imaging units) 11 to 17 being seven is an example, and other numbers of cameras may be used.

[0014] The in-vehicle image processing device 20 includes an image acquisition unit 21, an image storage unit 22, a subject recognition processing unit 23, an image adjustment unit 24, an image correction unit 25, an image selection unit 26, and a stereo recognition processing unit 27. The image acquisition unit 21 performs image acquisition processing on image signals captured by the seven cameras 11 to 17. The image signals of each camera acquired by the image acquisition unit 21 are supplied to the image storage unit 22. The image storage unit 22 temporarily stores the supplied captured images from the seven cameras 11 to 17.

[0015] The subject recognition processing unit 23 performs recognition processing on subjects (objects) around the vehicle from the images captured by the cameras 11 to 17, and outputs information of the recognized objects. When performing this recognition processing, information on the distance from the vehicle body to the recognized object is added based on information obtained by stereo recognition processing in the stereo recognition processing unit 27. The speed may be recognized from changes in distance. The image adjustment unit 24 performs adjustment processing to convert the images captured by the cameras 11 to 17 into appropriate images. This adjustment processing includes various types of image processing such as size adjustment and image brightness correction for properly performing selection by the image selection unit 26 and recognition by the stereo recognition processing unit 27. One of the adjustments performed by the image adjustment unit 24 is processing for detecting the amount of deviation between respective captured images.

[0016] The image correction unit 25 performs processing to correct the misalignment of each image based on the misalignment detection by the image adjustment unit 24. The image misalignment correction process includes, for example, correction of pitching, roll, and yaw misalignments, and specifically, vertical offset correction, horizontal offset correction, and angle correction. In this embodiment, since the stereo recognition processing unit 27 performs recognition processing in multiple sets, the system is configured to correct multiple images simultaneously.

[0017] In other words, as shown in Figure 2, the image correction unit 25 is configured to have multiple processing units, such as a first image correction unit 25a, a second image correction unit 25b, etc., and each image correction unit 25a, 25b, etc. performs different image corrections (first image correction processing, second image correction processing, third image correction processing, etc.). Specific examples of the correction processing performed by the image correction unit 25 will be described later in the explanation of Figure 8.

[0018] The image selection unit 26 performs image selection processing for stereo recognition using the image corrected by the image correction unit 25 and the image stored in the image storage unit 22. In this embodiment, since the stereo recognition processing unit 27 performs recognition processing in multiple sets, the system is configured to select multiple images simultaneously. In other words, the image selection unit 26 is also equipped with processing units that perform multiple selections, and each image selection unit is configured to select different images. A specific example of image selection by the image selection unit 26 will be described later in the explanation of Figure 5.

[0019] The stereo recognition processing unit 27 performs object recognition processing on two images selected by the image selection unit 26, and performs distance measurement processing to recognize the distance from the vehicle to the object based on the parallax between the two images. That is, as shown in Figure 3, the stereo recognition processing unit 27 also has a first stereo recognition processing unit 27a, a second stereo recognition processing unit 27b, a third stereo recognition processing unit 27c, a fourth stereo recognition processing unit 27d, etc., and is configured to perform multiple sets of stereo recognition processing simultaneously. However, in Figure 3, only the first stereo recognition processing unit 27a and the second stereo recognition processing unit 27b are shown, and the third stereo recognition processing unit and the fourth stereo recognition processing unit are not shown.

[0020] In this context, "simultaneous execution of stereo recognition processing" means that the processing is performed almost simultaneously. In addition to the configuration in which multiple processing systems perform processing in parallel simultaneously, as shown in Figures 2 and 3, it is also possible for a limited number of processing systems (for example, one stereo recognition processing unit) to sequentially perform the processing of each set of images in a time-division multiplexer within one frame period in which images are supplied from the camera. Therefore, the multiple image correction units and multiple stereo recognition processing units shown in Figures 2 and 3 are multiple in terms of the flow of image signals being processed, but they are not necessarily multiple in terms of hardware configuration. The distance information obtained from the distance measurement process is supplied to the subject recognition processing unit 23.

[0021] The in-vehicle image processing device 20 shown in Figure 1 can be configured as, for example, a computer. That is, as shown in the lower part of Figure 1, the in-vehicle image processing device 20 can be configured as a computer in which a CPU (Central Processing Unit) 20a, memory 20b, image input unit 20c, interface 20d, and image output unit 20e are connected by a bus line.

[0022] The CPU 20a is a processor that executes image recognition processing by running a program stored in memory 20b, thereby configuring various processing units in memory 20b. The processing units configured in memory 20b under the control of this CPU 20a are the subject recognition processing unit 23, image adjustment unit 24, image correction unit 25, image selection unit 26, and stereo recognition processing unit 27 mentioned above.

[0023] Memory 20b consists of a storage unit that acts as a work area for executing arithmetic processing under the control of the CPU 20a, and a storage unit that stores programs and various data. The image input unit 20c processes the images captured by cameras 11 to 17. Interface 20d exchanges information with other control devices (not shown) within the vehicle. For example, it transmits information about objects and distances around the vehicle, obtained by the in-vehicle image processing device 20, to other control devices. The image output unit 20e outputs the image processed by the in-vehicle image processing device 20 for display on a display unit (not shown) inside the vehicle.

[0024] The configuration of the in-vehicle image processing device 20, which performs calculations based on an implemented program, is just one example; other configurations are also possible. For example, part or all of the in-vehicle image processing device 20 may be implemented using dedicated hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0025] [Examples of camera field of view on a vehicle] Figure 4 shows examples of the field of view of the seven cameras 11 to 17 that supply captured images to the in-vehicle image processing device 20 in this embodiment. In Figure 4, the vehicle shown in the center has its front on the right and its rear on the left. Also, the top of Figure 4 is the left side, and the bottom of Figure 4 is the right side. The following describes the field of view of the seven cameras 11-17.

[0026] The first camera 11 is a front narrow-angle camera that captures images within the narrow front angle C1 range of the vehicle. Here, the narrow front angle is, for example, 30°. The first camera 11, which captures images within this narrow front angle C1 range, has the highest number of pixels among the seven cameras 11 to 17 and captures images with high resolution. The second camera 12 is a front wide-angle camera that captures the front wide-angle C2 range of the vehicle. Here, the front wide-angle is, for example, 120°. The second camera 12 that captures the front wide-angle C2 range has the same number of pixels as the first camera 11, or fewer pixels than the first camera 11, and captures images with the same resolution as the first camera 11, or lower resolution than the first camera 11.

[0027] The third camera 13 is a side-front camera that captures images of the left side front area C3 of the vehicle. Here, the side front is, for example, 120°. The fourth camera 14 is a side-front camera that captures images of the right side-front area C4 of the vehicle. Here, the side-front is, for example, 120°. The third camera 13 and the fourth camera 14 capture images with the same or fewer pixels as the first camera 11 and the second camera 12, and with the same or lower resolution as the first camera 11.

[0028] The fifth camera 15 is a side-rear camera that images the area C5 on the left side of the vehicle. Here, the side-rear is, for example, 120°. The sixth camera 16 captures images of the right side rear area C6 of the vehicle. Here, the side rear refers to a side rear camera with a range of, for example, 120°. The fifth camera 15 and the sixth camera 16 capture images with the same or fewer pixels as the first camera 11 and the second camera 12, and with the same or lower resolution as the first camera 11. Furthermore, when compared to the third camera 13 and the fourth camera 14, the fifth camera 15 and the sixth camera 16 capture images with the same or fewer pixels as the third camera 13 and the fourth camera 14, and with the same or lower resolution as the third camera 13 and the fourth camera 14. In addition, when compared to the seventh camera 17 (rear camera), which will be described next, the fifth camera 15 and the sixth camera 16 capture images with the same or fewer pixels as the seventh camera 17, and with the same or lower resolution as the seventh camera 17. The seventh camera 17 is a rear camera that captures images of the rear C7 area of ​​the vehicle. Here, the rear is, for example, 30°.

[0029] In this embodiment, seven cameras 11 to 17 capture images of the entire 360° horizontal surroundings of the vehicle. As shown in Figure 4, the imaging ranges C1 to C7 of each camera 11 to 17 overlap at least partially with the imaging ranges of the other cameras. In this embodiment, stereo processing is performed using this overlapping imaging range region.

[0030] [Examples of combinations for stereo processing] Figure 5 shows an example of an image that has undergone stereo processing. The image P1 captured by the first camera 11 at the front narrow angle C1 overlaps in its entirety with the image P2 captured by the second camera 12 at the front wide angle C2. The in-vehicle image processing device 20 uses almost the entire range P1a of the image P1 at the front narrow angle C1 and the approximately central range P2a of the image P2 at the front wide angle C2 to perform a set of stereo processing (first stereo recognition processing).

[0031] The image P2 captured by the second camera 12, which shows the front wide-angle C2, has a right half range P2b that overlaps with the left half range P4a of the side front C4 image P4 captured by the fourth camera 14. The in-vehicle image processing device 20 uses the right half range P2b of the front wide-angle C2 and the left half range P4a of the right side front C4 to perform stereo processing (second stereo recognition processing) on ​​a single pair.

[0032] Furthermore, the image P2 captured by the second camera 12 of the front wide-angle C2 has a left half range P2c that overlaps with the right half range P3a of the side front C3 image P3 captured by the third camera 13. The in-vehicle image processing device 20 uses the left half range P2c of the front wide-angle C2 and the right half range P3a of the left side front C3 to perform stereo processing (third stereo recognition processing) on ​​one set.

[0033] Furthermore, the rightmost range P4b of the image P4 of the right side front C4 of the fourth camera 14 overlaps with the leftmost range P6a of the image P6 of the side rear C6 of the sixth camera 16. The in-vehicle image processing device 20 uses the rightmost range P4b of the right side front C4 and the leftmost range P6a of the right side rear C6 to perform a set of stereo processing (fourth stereo recognition processing).

[0034] Furthermore, the leftmost range P3b of the image P3 of the left side front C3 captured by the third camera 13 overlaps with the rightmost range P5a of the image P5 of the left side rear C5 captured by the fifth camera 15. The in-vehicle image processing device 20 uses the leftmost range P3b of the left side front C3 and the rightmost range P5a of the left side rear C5 to perform a set of stereo processing (fifth stereo recognition processing).

[0035] Furthermore, a portion of the right edge P6b of the side rear C6 image P6 from the sixth camera 16 overlaps in imaging range with the left half P7b of the rear C7 image P7 from the seventh camera 17. The in-vehicle image processing device 20 uses the portion of the right edge P6b of the side rear C6 image P6 and the left half P7b of the rear C7 image P7 to perform a set of stereo processing (sixth stereo recognition processing).

[0036] Furthermore, a portion of the left edge P5b of the side rear C5 image P5 from the fifth camera 15 overlaps in imaging range with the right half P7a of the rear C7 image P7 from the seventh camera 17. The in-vehicle image processing device 20 uses the portion of the right edge P5b of the side rear C5 image P5 and the right half P7a of the rear C7 image P7 to perform a set of stereo processing (seventh stereo recognition processing).

[0037] Although not shown in Figure 5, the front narrow-angle image P1 and the side-front images P3 and P4 (C3 and C4) also overlap in some areas, so stereo processing may be performed using these images as well.

[0038] The in-vehicle image processing device 20 performs these stereo processing operations simultaneously. However, "simultaneously" here means that they are performed approximately at the same time, and they may be performed sequentially, for example, using time-division multiplexing. In order to perform stereo processing on each image, it is necessary to adjust the two images so that they can be properly stereo-processed. This involves using the image from one camera as a reference image and correcting the image from the other camera.

[0039] Figure 6 shows an example of a combination of a reference image and a correction target image when the in-vehicle image processing device 20 performs stereo image combination (camera combination). When performing stereo processing (first stereo recognition processing) with the image P1 of the front narrow-angle C1 and the image P2 of the front wide-angle C2, the image P1 of the front narrow-angle C1 captured by the first camera 11 is used as the reference. Then, the image P2 of the front wide-angle C2 captured by the second camera 12 is used as the image to be corrected for stereo processing.

[0040] When performing stereo processing (processing other than the 1st to 7th stereo recognition processing) using the front narrow-angle C1 image P1 and the side-front C3 and C4 images P3 and P4, the front narrow-angle C1 image P1 captured by the first camera 11 is used as the reference. Then, the side-front C3 and C4 images P3 and P4 captured by the third camera 13 and the fourth camera 14 are used as the images to be corrected for stereo processing. However, stereo processing combining the front narrow-angle C1 image P1 and the side-front C3 and C4 images P3 and P4 does not have to be performed.

[0041] When performing stereo processing (second stereo recognition processing and third stereo recognition processing) using the front wide-angle C2 image P2 and the side-front C3 and C4 images P3 and P4, the front wide-angle C2 image P2 captured by the second camera 12 is used as the reference. Then, the side-front C3 and C4 images P3 and P4 captured by the third camera 13 and fourth camera 14 are used as the images to be corrected for stereo processing.

[0042] When performing stereo processing (sixth stereo recognition processing and seventh stereo recognition processing) using the rear C7 image P7 and the side rear C5 and C6 images P5 and P6, the rear C7 image P7 captured by the seventh camera 17 is used as the reference. Then, the side rear C5 and C6 images P5 and P6 captured by the fifth camera 15 and sixth camera 16 are used as the images to be corrected for stereo processing.

[0043] When performing stereo processing (fourth stereo recognition processing and fifth stereo recognition processing) using the side-front images P3 and P4 of C3 and C4 and the side-rear images P5 and P6 of C5 and C6, the side-front images P3 and P4 captured by the fifth camera 15 and the sixth camera 16 are used as the reference. Then, the side-rear images P5 and P6 captured by the fifth camera 15 and the sixth camera 16 are used as the images to be corrected for stereo processing.

[0044] [Example of correction process] Figure 7 shows the process of obtaining a reference image and a corrected image from the captured images when performing the first stereo recognition process using the image P1 from the front narrow-angle C1 and the image P2 from the front wide-angle C2. In the first stereo recognition process, the image P1 of the front narrow-angle C1 is used as the reference, so the captured image P1 becomes the image for stereo processing, and almost the entire range of the captured image P1 becomes one of the images P1a when stereo processing is performed.

[0045] Then, the image P2 from the front wide-angle C2 is the image to be corrected in the first stereo recognition process, so offset correction and angle correction are performed so that the range P2a of the extracted image becomes the same as the range of image P1a. In addition, it is corrected so that it can be treated as an image of the same size as image P1a. When matching the size, for example, the number of pixels in the range P2a of image P2 may be increased by interpolation.

[0046] Figure 8 shows specific examples of offset correction and angle correction performed by the image correction unit 25. For example, as shown in the upper part of Figure 8, suppose that when performing stereo processing, an offset of a predetermined amount ΔY occurs in the vertical direction between the two images PL1 and PR1. In this case, the reference image (image PL1 in the example of Figure 8) is not corrected, and roll correction is performed on the image PR1 to be corrected, shifting it vertically by a predetermined amount ΔY. Although not shown in the diagram, when there is a left-right shift between the two images, the reference image PL1 is not corrected, and pitching correction is performed on the image to be corrected PR1 by a predetermined amount ΔX in the left-right direction. Furthermore, as shown in the lower part of Figure 8, assume that there is an angular shift of a predetermined amount ΔR between the two images PL2 and PR2 when performing stereo processing. In this case, the reference image (image PL2 in the example of Figure 8) is not corrected, and yaw shift correction is performed by rotating the image PR2 to be corrected by a predetermined amount ΔR.

[0047] [Effects of the processing in this embodiment] As explained above, when performing stereo processing with multiple cameras mounted on the vehicle, errors do not accumulate because a reference image and an image to be corrected are determined as shown in Figure 6. In other words, in the first stereo recognition process, by using the image captured by the first camera 11 (front narrow-angle camera), which has a high resolution, as a reference, object recognition and distance measurement can be performed with good stereo processing. The second and third stereo recognition processes also use the image captured by the second camera 12 as a reference, enabling good stereo processing. Furthermore, since the fourth and fifth stereo recognition processes also use the captured image from the third camera 13 or the fourth camera 14 as a reference, good stereo processing can be achieved.

[0048] On the other hand, the sixth and seventh stereo recognition processes can also perform object recognition and distance measurement with good stereo processing by using the image captured by the seventh camera 17 (rear camera), which has a higher resolution, as a reference. Furthermore, in each stereo processing step, the image captured by the camera is used as the reference image for one of the images, and only the other image is corrected. This reduces the number of images temporarily stored in the image processing device 20 for correction, thus reducing the burden on the image processing device 20 during image processing.

[0049] [Comparative example when the processing of this embodiment is not performed] To demonstrate the effect that errors do not accumulate when performing stereo processing in this embodiment, Figure 9 shows the processing in the case of conventional stereo processing for multiple sets. The example in Figure 9 shows a case where the reference images for stereo processing are sequentially selected from one image to the other. That is, when performing the first stereo recognition processing with the front narrow-angle image C1 and the front wide-angle image C2, the front wide-angle image C2 is corrected for pitch ΔX1, roll ΔY1, and yaw Δθ1. Furthermore, when performing the third stereo recognition process using the front wide-angle C2 image and the side-front C4 image, the side-front C4 image is corrected for pitch ΔX2, roll ΔY2, and yaw Δθ2, using the front wide-angle C2 image, which has been corrected in the first stereo recognition process, as a reference.

[0050] When performing the fifth stereo recognition process using the side-front C4 image and the side-rear C6 image, the side-rear C6 image is corrected for pitch ΔX3, roll ΔY3, and yaw Δθ3, using the side-front C4 image, which has been corrected in the third stereo recognition process, as a reference.

[0051] In this way, corrections are sequentially performed on adjacent field-of-view images around the vehicle, and the corrections are carried out sequentially on the images around the vehicle. When the processing cycle is completed and the second stereo recognition processing is performed on the front wide-angle C2 image and the side-front C3 image, the side-front C3 image is used as a reference, and pitch ΔX7, roll ΔY7, and yaw Δθ7 corrections are applied to the front wide-angle C2 image.

[0052] During this second stereo recognition process, the corrections ΔX7, ΔY7, and Δθ7 are applied to the front wide-angle C2 image, using the corrected side-front C3 image (which has accumulated corrections from previous stereo processes) as a reference. As a result, the range used for the front wide-angle C2 image becomes narrower than the range used for the third stereo recognition process. In other words, corrections are accumulated to perform multiple stereo processes, resulting in the use of a lower-resolution image and an inability to effectively utilize the pixel count of the camera's image sensor. In contrast, in this embodiment, image correction is not performed cumulatively, and stereo processing that effectively utilizes the number of pixels in the camera's image sensor is achieved.

[0053] [Differentiation] It should be noted that the embodiments described so far are detailed explanations provided to facilitate understanding of the present invention, and are not necessarily limited to those comprising all the configurations described. For example, the range captured by the seven cameras shown in Figure 5 is just one example, and the number of cameras and their respective imaging angles may be in other configurations.

[0054] Furthermore, in the example shown in Figure 5, the entire area around the vehicle is captured by seven cameras without any blind spots. However, it is also possible to omit some cameras and configure the system so that only a portion of the area around the vehicle is captured. In this case, for example, one system for stereo processing of a first group of cameras based on the front camera (front narrow-angle camera or front wide-angle camera) and another system for stereo processing of a second group of cameras based on the rear camera can be prepared. In other words, by prioritizing the front or rear camera, which has a relatively high resolution, as the reference image and correcting the side camera, stereo processing that effectively utilizes the resolution of the cameras, similar to the embodiment example described above, can be performed.

[0055] Furthermore, the configuration diagram shown in Figure 1 only shows control lines and information lines that are deemed necessary for explanation, and does not necessarily show all control lines and information lines in the actual product. In reality, it is safe to assume that almost all components are interconnected.

[0056] Furthermore, the in-vehicle image processing device 20 shown in Figure 1 is an example in which each processing unit is configured on a computer through program execution. In this case, the program may be stored on an external recording medium such as memory, an IC card, an SD card, or an optical disc and transferred to the computer (in-vehicle image processing device 20). [Explanation of symbols]

[0057] 11... Camera 1 (front narrow-angle camera), 12... Camera 2 (front narrow-angle camera), 13... Camera 3 (front side camera), 14... Camera 4 (front side camera), 15... Camera 5 (side rear camera), 16... Camera 6 (side rear camera), 17... Camera 7 (rear camera) 20...Automotive image processing unit, 20a...CPU, 20b...Memory, 20c...Image input unit, 20d...Interface, 20e...Image output unit, 21...Image acquisition unit, 22...Image storage unit, 23...Subject recognition processing unit, 24...Image adjustment unit, 25...Image correction unit, 26...Image selection unit, 26b...Second image selection unit, 27...Stereo recognition processing unit

Claims

1. An in-vehicle image processing device that arranges multiple cameras to capture the entire circumference of a vehicle, and performs stereo recognition at the point where the shooting ranges of adjacent cameras overlap, wherein the shooting ranges of each camera overlap. The plurality of cameras include at least a front narrow-angle camera that captures a narrow angle of the front of the vehicle, a front wide-angle camera that captures a wide angle of the front of the vehicle, a left front side camera that captures the left front side of the vehicle, and a right front side camera that captures the right front side of the vehicle. A first image correction unit performs image shift correction of the front wide-angle camera based on the image of the front narrow-angle camera, A first stereo recognition processing unit performs stereo recognition processing using the image captured by the front narrow-angle camera and the image captured by the front wide-angle camera that has been corrected by the first image correction unit. A second image correction unit performs image misalignment correction of the left front side camera based on the image of the front wide-angle camera, A second stereo recognition processing unit performs stereo recognition processing using the image captured by the front wide-angle camera and the image captured by the left front side camera, which has been corrected by the second image correction unit. A third image correction unit performs image misalignment correction of the right front side camera based on the image of the front wide-angle camera, An in-vehicle image processing device comprising: a third stereo recognition processing unit that performs stereo recognition processing using an image captured by the front wide-angle camera and an image captured by the right front side camera corrected by the third image correction unit.

2. The aforementioned plurality of cameras further include a rear camera for photographing the rear of the vehicle, a left side rear camera for photographing the left side rear of the vehicle, and a right side rear camera for photographing the right side rear of the vehicle. A fourth image correction unit performs image misalignment correction on the left side rear camera, which photographs the left side rear of the vehicle, based on the image of the rear camera that photographs the rear of the vehicle, A fourth stereo recognition processing unit performs stereo recognition processing using the image captured by the rear camera and the image captured by the left side rear camera, which has been corrected by the fourth image correction unit. A fifth image correction unit performs image misalignment correction of the right side rear camera based on the image of the aforementioned rear camera, A fifth stereo recognition processing unit performs stereo recognition processing using the image captured by the rear camera and the image captured by the right side rear camera, which has been corrected by the fifth image correction unit. A sixth image correction unit performs image misalignment correction of the left side rear camera based on the image of the left front side camera, A sixth stereo recognition processing unit performs stereo recognition processing using the image captured by the left front side camera and the image captured by the left side rear camera that has been corrected by the sixth image correction unit. A seventh image correction unit performs image misalignment correction on the right side rear camera, which captures the right side rear, using the image from the right front side camera as a reference. The system includes a seventh stereo recognition processing unit that performs stereo recognition processing using the image captured by the right front side camera and the image captured by the right side rear camera, which has been corrected by the seventh image correction unit. The in-vehicle image processing apparatus according to claim 1.

3. An image processing method for in-vehicle equipment, which involves arranging multiple cameras to capture the entire circumference of a vehicle, overlapping the shooting range of each camera with that of an adjacent camera, and performing stereo recognition at the point where the shooting ranges overlap, The plurality of cameras include at least a front narrow-angle camera that captures a narrow angle of the front of the vehicle, a front wide-angle camera that captures a wide angle of the front of the vehicle, a left front side camera that captures the left front side of the vehicle, and a right front side camera that captures the right front side of the vehicle. A first image correction process that corrects image misalignment of the front wide-angle camera based on the image of the front narrow-angle camera, A first stereo recognition process that performs stereo recognition processing using the image captured by the front narrow-angle camera and the image captured by the front wide-angle camera that has been corrected by the first image correction process, A second image correction process that corrects the image misalignment of the left front side camera based on the image of the front wide-angle camera, A second stereo recognition process that performs stereo recognition processing using the image captured by the front wide-angle camera and the image captured by the left front side camera that has been corrected by the second image correction process, A third image correction process that corrects the image misalignment of the right front side camera based on the image of the front wide-angle camera, The process includes a third stereo recognition process that performs stereo recognition using the image captured by the front wide-angle camera and the image captured by the right front side camera, which has been corrected by the third image correction process. Image processing method for in-vehicle equipment.

4. The aforementioned plurality of cameras further include a rear camera for photographing the rear of the vehicle, a left side rear camera for photographing the left side rear of the vehicle, and a right side rear camera for photographing the right side rear of the vehicle. A fourth image correction process that corrects the image misalignment of the left side rear camera based on the image of the aforementioned rear camera, A fourth stereo recognition process that performs stereo recognition processing using the image captured by the aforementioned rear camera and the image captured by the left side rear camera that has been corrected by the fourth image correction process, A fifth image correction process that corrects the image misalignment of the right side rear camera based on the image of the aforementioned rear camera, A fifth stereo recognition process that performs stereo recognition processing using the image captured by the aforementioned rear camera and the image captured by the right side rear camera that has been corrected by the fifth image correction process, A sixth image correction process is performed to correct the image misalignment of the left side rear camera, using the image of the left front side camera as a reference. A sixth stereo recognition process that performs stereo recognition processing using the image captured by the left front side camera and the image captured by the left side rear camera that has been corrected by the sixth image correction process, A seventh image correction process, which corrects the image misalignment of the right-side rear camera based on the image of the right-side front camera, The seventh stereo recognition process includes performing stereo recognition processing using the image captured by the right front side camera and the image captured by the right side rear camera, which has been corrected by the seventh image correction process. Image processing method for in-vehicle equipment according to claim 3.

Images