Camera system

Abstract

A camera system according to the present invention is provided with: a first camera group including a plurality of cameras disposed so as to be capable of capturing an image of the outside of the entire periphery that is continuous in at least one direction around a host vehicle; a second camera group that includes a plurality of cameras that are disposed so as to be able to capture an image of the outside of the entire periphery that is continuous in at least one direction around the own vehicle, and that are disposed so as to have a three-dimensional region that is an image capture field that overlaps at least the image capture field of the first camera group; and a stereo matching processing unit that obtains parallax information by performing stereo matching processing using an image captured by the first camera group and an image captured by the second camera group.

Description

Technical Field

[0001] The present invention relates to a camera system mounted on a vehicle to capture the entire surrounding area. Background Art

[0002] In the case of autonomous driving of a vehicle, it is necessary to use a camera mounted on the vehicle to sense the entire surroundings of the vehicle and obtain high-precision and high-density distance information.

[0003] In this regard, Patent Document 1 proposes, for the purpose of improving the accuracy of distance measurement, "a camera control device comprising: a detection unit, which detects the distance of an observation point within a detection range around a vehicle; and a correction unit, which corrects the detected distance of the observation point based on an overlap of the range of errors of the distances of multiple observation points within the detection ranges corresponding to the multiple detection units, that is, an overlap of the observation points. When an overlap of multiple observation points is detected, the correction unit corrects the detected distance of the observation point based on an overlap of the observation points close to the vehicle or an overlap of the observation points close to the observation points." Prior art literature Patent Literature

[0004] Patent Document 1: Japanese Patent No. 6819681 Summary of the invention Problem that the invention aims to solve

[0005] According to Patent Document 1, a configuration is adopted in which ultra-wide-angle cameras are installed around the vehicle or a configuration in which wide-angle stereo cameras are installed around the vehicle.

[0006] When using the stereoscopic vision of the ultra-wide-angle camera, there are problems such as reduced stereo matching accuracy due to the large difference in optical axes between the cameras that constitute the stereoscopic vision, reduced stereo matching accuracy at the edge of the field of view and in the distance due to lens distortion, and increased occlusion (blind spots) due to the reflection of vehicles.

[0007] Furthermore, when wide-angle stereo cameras are arranged around the vehicle, there are problems such as reduced stereo matching accuracy at the edge of the field of view, low designability (degree of freedom in product mounting), and increased costs.

[0008] Therefore, an object of the present invention is to provide a camera system that can construct multiple stereoscopic camera pairs using overlapping areas of multiple cameras to obtain distance information of the entire surroundings. Technical means of solving problems

[0009] Therefore, one form of the present invention is "a camera system comprising: a first camera group, which includes a plurality of cameras, and the plurality of cameras are configured to capture the entire surrounding outside world that is continuous in at least one direction around the own vehicle; a second camera group, which includes a plurality of cameras, and the plurality of cameras are configured to capture the entire surrounding outside world that is continuous in at least one direction around the own vehicle, and are configured to have a camera field of view that at least overlaps with the camera field of view of the first camera group, i.e., a stereo area; and a stereo matching processing unit, which uses the image captured by the first camera group and the image captured by the second camera group to perform stereo matching processing to obtain disparity information." Effects of the Invention

[0010] According to the present invention, it is possible to provide a camera system that can obtain distance information of the entire surroundings by forming a plurality of stereoscopic camera pairs using overlapping areas of a plurality of cameras.

[0011] Other problems, structures, and effects than those described above will become clear from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This figure shows a hypothetical example 1 for clarifying the problem of continuously measuring the distance to the entire surroundings. Figure 2 This figure shows a hypothetical example 2 for clarifying the problem of continuously measuring the distance to the entire surroundings. Figure 3 This figure shows a hypothetical example 3 for clarifying the problem of continuously measuring the distance to the entire surroundings. Figure 4 A diagram showing the basic concept of camera configuration according to the present invention. Figure 5 A diagram showing the basic concept of the three-dimensional region of the present invention. Figure 6 1 is a diagram showing a configuration example of a 360-degree multi-stereoscopic camera system 1 composed of six cameras. Figure 7 A diagram showing a configuration example of a 360-degree multi-stereoscopic camera system 2 composed of six cameras. Figure 8 A diagram showing a configuration example of a 360-degree multi-stereoscopic camera system 1 composed of seven cameras. Fig. 9 A diagram showing a configuration example of a 360-degree multi-stereoscopic camera system 2 composed of seven cameras. Fig.10 To enlarge Figure 7 A view of the area formed by a 360-degree multi-stereo camera system. Fig.11A diagram showing an example of the configuration of a 360-degree multi-stereo camera system consisting of 10 cameras. Fig.12 For the general Fig.11 and Fig.13 A diagram that summarizes the idea. Fig.13 This figure shows a configuration example of two groups of four cameras. Fig.14 This figure shows an example of a configuration that emphasizes the rear. Fig.15 This figure shows an example of a configuration that emphasizes cost. Fig.16 A diagram showing other configuration examples. Fig.17 This is a diagram generally showing the response of Example 2. Fig.18 1 is a diagram showing a configuration example of a vehicle-mounted imaging device 10 that processes image information from a plurality of cameras. Fig.19 1 is a diagram showing a configuration example of a vehicle-mounted imaging device 10 that processes image information from a plurality of cameras. Fig. 20 It is a diagram showing an example of a priority standard when three-dimensional regions overlap. Fig.21 This is a diagram showing the concept of priority criterion 1. Fig. 22 This is a diagram showing the concept of priority criterion 2. Fig.23 This is a diagram showing the concept of priority criterion 3. Fig.24 This is a diagram showing the concept of priority criterion 4. Fig.25 This is a diagram showing the concept of priority criterion 5. Fig.26 This is a diagram showing the concept of priority criterion 6. Fig. 27 : is a diagram showing an improved point for obtaining high-precision disparity information. Fig.28 To express Fig.18 , Fig.19 FIG. 1 is a diagram showing the processing contents of the stereo matching priority determination unit 15 . Fig.29 This figure shows a configuration example in which four cameras are added to the camera group A, namely, front, back, left, and right. Fig.30 This figure shows a configuration example in which four cameras in the front, rear, left and right directions are added to the camera group A1. Fig.31This figure shows a configuration example in which four cameras are added to the camera group A2. Fig.32 This is a diagram showing the arrangement of four-surrounding cameras of the vehicle in a layout in which 12 cameras are added to the camera group A2 in the front, rear, left and right directions. Fig.33 The figure shows the arrangement of cameras on all four sides of the vehicle in a layout in which 10 cameras are added to the camera group A in the front, back, left and right directions. Fig.34 A diagram showing the arrangement of four-surrounding cameras of the vehicle in camera group A2. Fig.35 A diagram showing the arrangement of four cameras on the vehicle in camera group A1. DETAILED DESCRIPTION

[0013] Next, embodiments of the present invention will be described. Example 1

[0014] In the first embodiment, a case is described in which a plurality of stereoscopic camera pairs are formed using overlapping areas of the fields of view of a plurality of cameras, and the distance to the entire surroundings is continuously measured when acquiring distance information of the entire surroundings.

[0015] However, before entering into the description of the embodiments of the present invention, several hypothetical cases are studied to clarify the problem of continuously measuring the distance to the entire surroundings. Figure 1 , Figure 2 Cases 1 and 2 are presented, which are examples of configuring fisheye cameras (HFOV 180 degrees or more) in four directions, namely, front, back, left, and right, of a vehicle 1 to form a surround camera system for monitoring the entire surroundings of the vehicle (360 degrees in the horizontal direction).

[0016] Figure 1 This is an example in which fisheye cameras are arranged in four directions, front, back, left, and right, of the vehicle 1, and the monitoring areas E of each camera are formed in the front E1, rear E2, left E3, and right E4. Therefore, the areas RE of each camera overlap to form the stereoscopic field of view areas (stereoscopic areas RE1, RE2, RE3, RE4) in four directions inclined relative to the vehicle.

[0017] Figure 2 This is an example where fisheye cameras are arranged in four directions at the four corners of the vehicle 1, and the monitoring area E of each camera is formed in the left front E1, the right rear E2, the left rear E3, and the right front E4. Therefore, the area E of each camera overlaps to form a stereoscopic field of view (stereoscopic areas RE1, RE2, RE3, RE4) in the four directions of front, back, left, and right with respect to the vehicle 1.

[0018] according to Figure 1 , Figure 2The following facts can be understood from the configuration example. The first point is that the stereoscopic area will have gaps on all sides that are equivalent to the full width and length of vehicle 1, so a blind spot area of ​​the stereoscopic field of view will be generated. The second point is that the optical axis angle difference between the cameras forming the stereoscopic area is large (90 degrees), so the accuracy of stereo matching is reduced. In other words, there is a large difference in the direction of the optical axes between the two cameras that make up the stereo pair, so stereo matching is difficult. The third point is that the stereoscopic area contains the area at the edge of the image of the ultra-wide-angle camera, so the influence of distortion and chromatic aberration is large, resulting in a problem of reduced ranging accuracy.

[0019] Figure 3 The hypothetical case 3 is shown, with the right side showing the four views of the vehicle 1 (front and rear surfaces, side surfaces, and top surface), and the left side showing the final three-dimensional area. In this example, a 360-degree multi-stereo camera is used to capture the entire surroundings of the vehicle (360 degrees in the horizontal direction) by arranging four fisheye cameras (HFOV 180 degrees or more) on the roof of the vehicle. Two fisheye cameras are arranged on the roof for the front and rear, respectively, to form three-dimensional areas in the front and rear.

[0020] according to Figure 3 The following facts can be understood from the configuration example. First, the imaging range of one camera is wide, so it is okay to have fewer cameras, but the angular resolution of the image is reduced, making it difficult to measure distances at a distance, and the performance of object recognition as a monocular camera is also reduced. Second, the ultra-wide-angle camera has a large depth of field (wide focusing range), and when it is configured outdoors, such as on the roof of a vehicle, it is easy for the visibility to be reduced due to lens attachments such as water droplets and dirt, and reliability is particularly required in the front area, so it is not ideal. Third, although there is no gap in the three-dimensional area, the camera must be mounted on the ceiling, resulting in a large shielding area due to the reflection of the vehicle roof into the lower part of the image.

[0021] according to Figure 1 , Figure 2 , Figure 3 The above results in the hypothetical example, the basic idea of ​​the present invention is to construct a camera system as follows. In this basic idea, the first point is like Figure 4 As shown in FIG. 1 , a plurality of cameras C are installed at the peripheral part of the vehicle 1 where the reflection of the vehicle body is small. In this case, the camera C is a monocular camera composed of a lens and an imaging element, or a camera composed of a lens with a curved surface shape such as a sphere, hyperbola, cone, parabola, etc. and an imaging element, and the peripheral part is not limited to the interior or exterior of the vehicle. However, it is preferable to arrange the camera at a position where the reflection of the vehicle body is small.

[0022] The second point is like Figure 5As shown in FIG. 1 , multiple monocular cameras C are combined to form a stereoscopic area of ​​the entire surrounding. The cameras are arranged in such a way that the stereoscopic area of ​​the entire surrounding does not have a gap larger than the width of the vehicle. In other words, Figure 5 The ends of the three-dimensional regions are brought into contact or intersecting as shown.

[0023] The third point is to select Figure 5 As shown, the angular difference in the optical axis direction between the cameras forming the stereo pair is made less than 90 degrees. This can prevent the reduction in distance measurement accuracy caused by the optical axis angular difference.

[0024] Figure 4 An example of a camera system is a system of seven cameras, including two front cameras CF for shooting the front, two left and right cameras (CR, CL) for shooting the left and right, and one rear camera CB for shooting the rear. The left and right cameras (CR, CL) also include cameras that shoot sideways, obliquely, and frontally.

[0025] also, Figure 4 An example of a configuration is a camera system consisting of 7 cameras, namely, two front cameras CF with a field of view of 120 degrees, a left front camera CLF with a field of view of 90 degrees, a right front camera CRF, a left rear camera CLR with a field of view of 120 degrees, and a right rear camera CRR as side cameras on the left and right sides, and a rear camera CB with a field of view of 60 degrees as a rear camera. In this way, stereoscopic vision can be formed by shooting in the front, back, left and right directions with more than two monocular cameras. Furthermore, the condition here is that as long as the stereoscopic field of view can be ensured in the entire surrounding, the camera can also use a fisheye lens, so it can also be Figure 4 The number of cameras in the composition is less than or greater than the number of cameras.

[0026] In addition, here is a rule for marking the symbol C representing the camera, that is, F, B, R, and L representing the front, back, left, and right are marked after the camera C. Among them, the front and back are given priority. Therefore, for the left and right cameras, the optical axis direction is obliquely forward and is distinguished by F, and the optical axis direction is obliquely backward and is distinguished by B. In addition, for multiple cameras of the same type, when it is impossible to distinguish them by the above method, they are distinguished by numbers as needed. In addition, this rule for marking symbols also applies to the symbol RE representing the field of view area.

[0027] In the same manner, the optical axes of cameras and the "directions" of the areas in this specification are first defined in terms of front, back, left, and right and are recorded as front camera, rear camera, right camera, and left camera. However, sometimes left and right cameras are collectively referred to as side cameras. In addition, regarding front, back, left, and right, left and right are used before front and back. Therefore, the left and right cameras with their optical axes facing forward are called side front cameras (right front camera, left front camera), and the left and right cameras with their optical axes facing backward are called side rear cameras (right rear camera, left rear camera).

[0028] Figure 5 To indicate by Figure 4 A diagram of a three-dimensional area formed by a system of seven cameras is shown.

[0029] like Figure 5 As shown, a 120-degree stereoscopic area REF is ensured in the front by two 120-degree front cameras CF, 90-degree stereoscopic areas RER and REL are ensured on the left and right sides by a combination of a left front camera CLF and a left rear camera CLR as well as a right front camera CRF and a right rear camera CRR, and a 60-degree (30 degrees and 30 degrees) stereoscopic area REB is ensured in the rear by a combination of a rear camera CB and a left rear camera CLB and a right rear camera CRB.

[0030] Figure 5 The three-dimensional area of ​​the entire surrounding shown has no gaps equivalent to the width of the vehicle because the ends of each three-dimensional area are in contact or intersecting. In other words, a continuous three-dimensional area is formed across the entire surrounding of the vehicle. In other words, it can be called a camera system, comprising: a first camera group, which is configured to capture the entire surrounding environment around the vehicle in at least one direction (for example, horizontal direction); a second camera group, which is composed of a plurality of cameras having a camera area that at least overlaps with the camera area of ​​the first camera group; and a parallax calculation unit, which uses the image captured by the first camera group and the image captured by the second camera group to perform stereo matching processing to obtain parallax information.

[0031] In addition, according to Figure 5 The camera system configuration can achieve the following effects: the vertical shading area caused by the reflection of the vehicle body is reduced compared to the assumed case 3, and distance information can be obtained from the entire surrounding area in the horizontal direction. There is no need to use an ultra-wide-angle lens such as a fisheye lens, so the distance measurement accuracy is high in the distance.

[0032] Hereinafter, a camera system capable of continuously measuring the distance to the entire surroundings will be referred to as a 360-degree multi-stereo camera system. A configuration example for realizing the 360-degree multi-stereo camera system will be described in detail later in relation to the number of cameras.

[0033] First, as one example of a configuration using six cameras, Figure 6 A 360-degree multi-stereoscopic camera system using two sets of three cameras for capturing images of the front, rear left, and rear right will be described. Figure 6 The upper left portion of FIG. 1 shows a camera group 1 including three cameras for shooting the front, the left rear, and the right rear. The three cameras of the camera group 1 all have a field of view of 120 degrees.

[0034] Figure 6 The lower left portion of the figure shows a camera group 2 including three cameras for shooting the front, left rear, and right rear. As in the case of camera group 1, the three cameras of camera group 2 each have a field of view of 120 degrees. The camera groups 1 and 2 are configured so that the field of view FOV of each camera do not overlap, and the horizontal left and right field of view HFOV is configured to be a total of 360 degrees.

[0035] Figure 6 The right part of shows the stereoscopic area obtained by combining camera group 1 and camera group 2. The camera area of ​​camera group 1 overlaps with the camera area of ​​camera group 2, and the overlapping camera area is called the stereoscopic area. The stereoscopic areas REF, RERB, and RELB in the front and left and right rear are 120 degrees respectively, so it is equivalent to arranging three parallel stereo cameras with a field of view of 120 degrees on the vehicle in a manner without overlapping the field of view. However, in this method, a blind spot area X will be generated directly behind the vehicle. In addition, the adjacent parts of the stereoscopic area become the ends of the field of view of each camera, so an area Y is generated where the recognition accuracy of the monocular camera is reduced.

[0036] That is to say, depending on the installation conditions of the camera, there is a blind spot area X (diagonal line portion) that is blocked by a part of the vehicle body and cannot be photographed by the monocular camera. Therefore, it is preferred to be able to perform 360-degree seamless shooting in the form of at least a monocular area. In addition, when using a wide-angle camera, the influence of distortion and chromatic aberration on the peripheral part of the field of view angle is large, and the object recognition performance of the monocular camera is reduced, and the object recognition performance sometimes deviates. Therefore, in order to suppress the deviation of object recognition performance, it is preferred to configure multiple cameras in a manner that does not produce an area that is only photographed in the peripheral part of the field of view angle, that is, an area Y where the recognition accuracy of the monocular camera is reduced. Therefore, it is preferred to be able to shoot 360 degrees with multiple monocular cameras with different optical axis directions.

[0037] As a second configuration example using six cameras, use Figure 7A 360-degree multi-stereoscopic camera system consisting of two groups: group 1 of three cameras that capture the front, left rear, and right rear, and group 2 of three cameras that capture the left front, right front, and rear. Figure 7 The upper left portion shows a group 1 of three cameras that capture the front, the left rear, and the right rear. The three cameras of the camera group 1 have a field of view of 120 degrees + α1 or 120 degrees + α2. Figure 7 The lower left part shows a group 2 of three cameras that shoot the left front, right front, and rear. As in the case of camera group 1, the three cameras of camera group 2 have a field of view of 120 degrees + α3 or 120 degrees + α4. In addition, α1+α2+α3+α4 is an added value of the field of view of each camera, and is set to an arbitrary value greater than 0 in a manner such that the camera areas of adjacent cameras touch or intersect each other according to the installation conditions of each camera.

[0038] The camera group 1 and the camera group 2 are arranged so that at least the field of view FOV of each camera overlaps, that is, the ends of the field of view FOV intersect with each other. In other words, the total of the horizontal left and right field of view HFOV is 360 degrees + α1 + α2 + α3 + α4. Here, the ends of each three-dimensional area are in contact or intersected in such a way that the three-dimensional area where the camera areas of the camera group 1 and the camera group 2 overlap can cover the entire surroundings.

[0039] according to Figure 7 The optical axis directions of the six cameras are all different. Figure 6 The camera configuration is different. Figure 7 In the present embodiment, stereoscopic vision is performed by a pair of two cameras in the same direction within the field of view (for example, a camera of camera group 1 shooting the front and a camera of camera group 2 shooting the left front). In other words, stereoscopic vision is performed by a camera of camera group 1 and a camera of camera group 2 in the entire surrounding of the vehicle, and the stereoscopic vision area is more than 360 degrees in total. In the present configuration example shown later, the optical axes of the two cameras performing stereoscopic vision are different, but in the extended example, cameras with the same optical axis direction may be further included.

[0040] Figure 7 The right part shows the stereoscopic area obtained by combining the two camera groups 1 and 2 in this configuration example. In this configuration example, five stereoscopic areas are formed, namely the front stereoscopic area REF, the left and right stereoscopic areas RER, REL, and the rear stereoscopic areas RERB, RELB. In this case, the total HFOV of the stereoscopic area is also more than 360 degrees, and the total field of view angle of multiple monocular cameras can also cover 360 degrees without gaps.

[0041] This example is based on Figure 5 The configuration example of 7 cameras shown in the figure realizes 360-degree multi-stereoscopic cameras by reducing the number of cameras by 6. By reducing the number of cameras, the cost and image processing load can be reduced, which can be called a "cost-oriented type."

[0042] Next, as one example of a configuration using seven cameras, Figure 8 A 360-degree multi-stereoscopic camera system consisting of two groups: group 1 of three cameras that capture the front, left rear, and right rear, and group 2 of four cameras that capture the front, left front, right front, and rear. Figure 8 The upper left portion shows a group 1 of three cameras that capture the front, the left rear, and the right rear. The three cameras of the camera group 1 have a field of view of 120 degrees + α1 or 120 degrees + α2. Figure 8 The lower left portion shows a group 2 of four cameras that shoot the front, left front, right front, and rear. Of the four cameras in camera group 2, the cameras that shoot the left front and right front have a field of view of 120 degrees + α3, the camera that shoots the rear has a field of view of 90 degrees + α4, and the camera that shoots the front has a field of view of 30 degrees + α5.

[0043] Figure 8 The right part shows the stereoscopic area obtained by combining the two camera groups 1 and 2 in this configuration example. In this configuration example, seven stereoscopic areas are formed, namely, the front stereoscopic area REF, the left and right front stereoscopic areas RERF and RELF, the left and right stereoscopic areas RER and REL, and the left and right rear stereoscopic areas RERB and RELB. In this case, the total HFOV of the stereoscopic area is also more than 360 degrees, and the total field of view angle of multiple monocular cameras can also cover 360 degrees without gaps.

[0044] This configuration example uses a narrow-angle (30-degree) camera as a monocular camera for shooting the front. The narrow-angle camera can be used for small object detection in the distance (traffic light recognition, etc.). In this way, by combining the narrow-angle camera used for small object detection in the distance with the wide-angle (120-degree) camera, a 360-degree multi-stereo camera using the existing monocular camera system can be realized, which can be called "using the existing camera system type". According to this configuration example, it is possible to take into account both the performance of detecting objects far ahead and the performance of continuous stereo ranging in 360 degrees around the vehicle.

[0045] Next, as a second example of a configuration using seven cameras, Fig. 9 A 360-degree multi-stereoscopic camera system consisting of two groups: group 1 of three cameras that capture the front, left rear, and right rear, and group 2 of four cameras that capture the front, left side, right side, and rear. Fig. 9 The upper left portion shows a group 1 of three cameras that capture the front, the left rear, and the right rear. The three cameras of the camera group 1 have a field of view of 120 degrees + α1 or 120 degrees + α2. Fig. 9 The lower left portion of the image shows a group 2 of four cameras that shoot the front, left side, right side, and rear. Among the four cameras in the camera group 2, the camera that shoots the front has a field of view of 120 degrees + α1, the camera that shoots the rear has a field of view of 60 degrees + α4, and the cameras that shoot the left front and right front have a field of view of 90 degrees + α3.

[0046] Fig. 9 The right part shows the stereoscopic area obtained by combining the two camera groups 1 and 2 in this configuration example. In this configuration example, five stereoscopic areas are formed, namely the front stereoscopic area REF, the left and right stereoscopic areas RER, REL, and the left and right rear stereoscopic areas RERB, RELB. In this case, the total HFOV of the stereoscopic area is also more than 360 degrees, and it can cover 360 degrees without gaps when viewed from multiple monocular cameras.

[0047] In this configuration example, the front area with high importance constitutes a seamless stereo area, so high-precision parallax information can be obtained. In addition, the side area can also improve the distance accuracy by reducing the difference in the optical axis direction between the cameras that constitute the stereo pair. In this way, the area with high importance in the recognition of external information is stereoscopically viewed as a seamless stereo area, or the difference in the optical axis direction between the cameras that constitute the stereo pair is reduced. This can realize a 360-degree multi-stereo camera in which the accuracy in the area with high importance is emphasized, which can be called a "type that emphasizes accuracy in important areas." Furthermore, the area with high importance is not limited to the front of the vehicle, and any area such as the side, rear, and top of the vehicle can be regarded as an area with high importance.

[0048] Fig.10 To enlarge Figure 7 A diagram of a stereoscopic area formed by a 360-degree multi-stereoscopic camera system in the configuration example shown. In this case, a portion (shaded portion) that does not form a stereoscopic area appears in the vehicle vicinity. Fig.10 The total of the field of view areas at the predetermined positions of the camera group 1 and the camera group 2 shown in the upper and lower left part is 360 degrees, so there may be an area near the vehicle where a stereoscopic field of view cannot be ensured due to positional deviation of the camera arrangement locations.

[0049] Fig.10 The right side shows the stereoscopic viewing area (REF+RER+REL+RERB+RELB). The area outside the stereoscopic viewing area corresponds to the stereoscopic blind spot area Z, which is enlarged and shown in Fig.10Thus, the problem of the blind spot area Z of the stereoscopic field of view mainly occurs near the vehicle, does not affect the distance measurement at a distance, and the blind spot area is narrow, but an example of a structure for coping with this problem will be further shown below.

[0050] The problem arises from Figure 4 For the horizontal direction, the blind spot area Z of the stereoscopic field of view can be narrowed by configuring the cameras in such a way that the distance difference between the two cameras configured on the upper surface of the vehicle is reduced or by adjusting the field of view of each camera. Figure 1 and Figure 2 As shown, the blind spot area of ​​the stereoscopic field of view in the past was equivalent to the width of the vehicle, but in this configuration example, it can be narrowed to the distance between the cameras and can be limited to a short distance.

[0051] Next, a description will be given of further eliminating and narrowing the blind spot area of ​​the stereoscopic field of view. Fig.11 is a diagram showing an example of a configuration that can eliminate a blind spot area of ​​a stereoscopic field of view, and Figure 7 The difference between the examples of Figure 7 Based on the configuration example, 4 more cameras are added, front, back, left and right.

[0052] Fig.11 The upper left part shows the Figure 7 The camera group 1 and the camera group 2 shown in the figure are combined to form a structure consisting of Fig.11 This is achieved by a camera group A consisting of six cameras mounted on the right vehicle 1, namely, a front camera CF1, a right front camera CRF, a right rear camera CRB, a left front camera CLF, a left rear camera CLB, and a rear camera CB1.

[0053] Add to it Fig.11 The camera group 3 at the lower left eliminates the blind spot area of ​​the stereoscopic field of view described above. Fig.11 This is achieved by four fisheye cameras, namely, a front camera CF2, a right camera CR2, a left camera CL2, and a rear camera CB2, among the cameras mounted on the right vehicle 1.

[0054] Fig.11 The central part is Fig.11The stereoscopic area obtained by overlapping the field of view of the camera group A (a combination of camera group 1 and camera group 2) and camera group 3 shown on the left completely covers the entire area around the vehicle in the horizontal direction. By adding four fisheye cameras mounted on the front, back, left and right sides, it is possible to form a stereoscopic area in the vicinity with the camera of camera group A and the camera pair of the fisheye camera. This configuration example can be called a "full-range coverage type" that covers the entire surrounding area from the vicinity to the distant area with a stereoscopic area.

[0055] for Figure 7 For example, we can take Fig.11 The fisheye camera-based solution for the blind spot area of ​​the stereoscopic field of view shown in the previous section is as follows. For other configuration examples, add Fig.11 The camera group 3 in the lower left part is also effective. In addition, although four fisheye cameras are illustrated as the camera group 3 in this configuration example, the type and number of cameras are not limited to this.

[0056] Fig.12 For the general Fig.11 A diagram that summarizes the idea. Fig.12 The upper part shows an example of the arrangement of cameras on the vehicle 1. The example shows a situation where fisheye cameras CF, CR, CL, and CB are added to the camera group A mounted on the vehicle 1. Fig.12 The lower left portion shows the imaging area of ​​each camera and each stereo area separately in this case. Fig.12 The lower right shows the Fig.12 The arrangement example of the cameras shown in the upper part also includes the vicinity of the vehicle 1 and realizes a three-dimensional area in the entire surrounding area.

[0057] Although not shown, the plurality of camera images are introduced into and processed by a computing unit in the vehicle 1. Specifically, a parallax calculation is performed to obtain a parallax by stereo matching an overlapping region of any two images. Thus, a distance calculation to an object captured in the camera image is performed.

[0058] In summary, the 360-degree multi-stereo camera system described in Example 1 is a camera system having a plurality of cameras mounted on a vehicle and a parallax calculation unit, wherein the plurality of cameras are mounted on a vehicle, and the parallax calculation unit uses a plurality of images captured by the plurality of cameras to perform stereo matching processing to obtain parallax. In this case, an area including an external area in front of the vehicle is defined as a first area, an area including an external area on the right side of the vehicle is defined as a second area, an area including an external area on the left side of the vehicle is defined as a third area, and an area including an external area on the rear side of the vehicle is defined as a fourth area.

[0059] In addition, multiple cameras are arranged on the vehicle in the following manner, that is, in the first area to the fourth area, there are a first repeated camera field of view, a second repeated camera field of view, a third repeated camera field of view and a fourth repeated camera field of view as repeated camera fields of view in the horizontal direction of at least two cameras.

[0060] Regarding the obtained image, the end of the first repeated imaging field of view is at least partially adjacent to the end of the second repeated imaging field of view and the end of the third repeated imaging field of view, or at least partially overlaps with the second repeated imaging field of view and the third repeated imaging field of view, and the end of the fourth repeated imaging field of view is at least partially adjacent to the end of the second repeated imaging field of view and the end of the third repeated imaging field of view, or at least partially overlaps with the second repeated imaging field of view and the third repeated imaging field of view.

[0061] Furthermore, the first to fourth repeated imaging fields are respectively captured by at least two cameras having different optical axis directions, and the parallax calculation unit performs stereo matching processing using a plurality of images captured within the first to fourth repeated imaging fields. Example 2

[0062] In the first embodiment, the realization of the 360-degree multi-stereoscopic camera system is described, and the structure of the blind spot area that eliminates the stereoscopic field of view is also explained.

[0063] The problem in this case is that the failure of the camera of vehicle 1 will cause the loss of the stereoscopic area and the inability to measure the distance. The front camera is particularly important. Redundancy is an effective countermeasure for this problem, so the reliability improvement countermeasure when the sensor fails is described in Example 2. Furthermore, although the front reinforcement countermeasure is particularly shown, it can of course be applied to the side or rear as needed.

[0064] Fig.13 An example of a 360-degree multi-stereoscopic camera configuration formed by combining a camera group 1 and a camera group 2 using four cameras is shown in FIG. Fig.13 In the upper left and lower left parts of the image, the field of view of camera group 1 and camera group 2 are marked with the marks of the cameras that captured the area. Fig.13 The field of view of the camera group 1 on the upper left is basically Figure 7 The camera group 1 shown in the upper left part is formed by adding a narrow-angle (30-degree) front camera CF3.

[0065] Fig.13 The field of view of the camera group 2 at the lower left is basically Figure 7The camera group 2 shown in the lower left part is formed by adding a wide-angle (120-degree) front camera CF2. The cameras are arranged so that the field of view FOV partially overlaps and the total horizontal field of view HFOV is 360 degrees or more.

[0066] Fig.13 The center portion shows the stereoscopic area generated when camera group 1 and camera group 2 are combined. Here, the area where multiple stereoscopic areas overlap because the field of view areas of three or more cameras overlap is called an overlapping stereoscopic area. Fig.13 The portion shown in the central portion as repeated three-dimensional regions REX1, REX2, and REX3 corresponds to the repeated three-dimensional region.

[0067] Fig.13 The right part shows an example of camera configuration. The overlapping stereo area REX1 is covered by three cameras: CLF, CF1, and CF2; the overlapping stereo area REX2 is covered by three cameras: CF1, CF2, and CF3; and the overlapping stereo area REX3 is covered by three cameras: CRF, CF1, and CF2. Furthermore, the overlapping stereo area REX2 is covered by four cameras including either CLF or CRF in the close-range portion.

[0068] Obviously, the problem that the three-dimensional area is lost and the distance measurement cannot be performed due to the failure of the camera of the vehicle 1, which needs to be dealt with in the second embodiment, can be solved by taking measures especially for the important front. Fig.13 For example, if CF1 of the three cameras that capture the overlapping stereoscopic area REX1 fails, the stereoscopic area formed by the remaining CLF and CF2 can be ensured, so the stereoscopic vision of the area can continue even after the failure of CF1. Fig.13 The configuration example shown can be called a "forward redundancy-oriented type".

[0069] Fig.14 This shows an example of a configuration that emphasizes the rear. Fig.13 The difference of the configuration example shown is that a 120-degree rear camera CB2 is added instead of Fig.13 The front camera CF3 with a narrow angle is shown. In this case, overlapping stereoscopic areas REX4 and REX5 are formed in the rear. The overlapping stereoscopic area REX4 in the rear is covered by the three cameras CRB, CB1, and CB2, and the overlapping stereoscopic area REX5 is covered by the three cameras CLB, CB1, and CB2, so even if one of the three cameras fails, the stereoscopic areas formed by the remaining two cameras can be ensured. Fig.14 The configuration example shown can be called a "rearward redundancy-oriented type".

[0070] Fig.15This shows an example of a cost-conscious configuration. Fig.13 The difference is that in Figure 7 In the 360-degree multi-stereo camera composed of the camera group 1 and the camera group 2 using three cameras shown, a narrow-angle front camera CF3 is added to the camera group 2. In this case, overlapping stereo areas REX6 and REX7 are formed in the front. The overlapping stereo area REX6 in the front is covered by the three cameras CLF, CF1, and CF2, and the overlapping stereo area REX7 is covered by the three cameras CRF, CF1, and CF2, so even if one of the three cameras fails, the stereo area formed by the remaining two cameras can be ensured. Fig.15 The configuration example shown can be called a "cost-oriented redundancy type".

[0071] Fig.16 is a diagram showing other configuration examples, and Fig.13 The difference of the configuration example shown is that Fig.11 The combination of camera groups 1, 2, and 3 shown is a multi-stereo camera (camera group B) using 10 cameras with the addition of a narrow-angle (30-degree) front camera CF1 and a wide-angle (120-degree) front camera CF2. Fig.16 The configuration example of the cameras in the upper part shows various variations as the camera group B, and this embodiment can cope with all these variations. In this case, an overlapping three-dimensional area formed by three or more cameras is formed in the substantially entire area in front, so even if one of the three cameras fails, the three-dimensional area formed by the remaining two cameras can be ensured.

[0072] Fig.17 This is a diagram that summarizes the response of Example 2. First, Fig.17 The left part shows the situation where the entire surrounding is covered by multiple cameras to ensure stereoscopic vision. Among them, for the front, the wide-angle (120-degree) cameras CF1 and CF2 and the narrow-angle (30-degree) camera CF3 realize stereoscopic vision from long distance to medium and long distance. Moreover, for close distance, although other cameras also participate in multiplexing, the description here is omitted.

[0073] Fig.17 The upper right part shows the situation where the wide-angle camera CF2 fails in the stereoscopic vision formed by the three cameras at medium and long distances. Although the stereoscopic vision of the medium distance area using camera CF2 is lost, the presence of cameras CF1 and CF3 ensures the stereoscopic vision of the 30-degree range. Furthermore, the same is true when camera CF1 fails instead of camera CF2.

[0074] Fig.17The lower right part shows the situation where the narrow-angle camera CF3 fails in the stereoscopic vision formed by the three cameras at medium and long distances. Although the stereoscopic vision of the medium and long distance area using camera CF3 is lost, the presence of cameras CF1 and CF2 ensures the stereoscopic vision in a large range at close distances. Example 3

[0075] In the third embodiment, a case where highly accurate parallax information is acquired when the processing of the stereoscopic region described in the second embodiment is repeated will be described.

[0076] For example, Fig.17 The question is, when a part of the stereoscopic area formed by cameras CF1 and CF2 overlaps with the stereoscopic area formed by other cameras (for example, CF1 and CF3), which one of the stereoscopic area formed by cameras CF1 and CF2 and the stereoscopic area formed by cameras CF1 and CF3 should be used to generate disparity information.

[0077] Fig.18 This is a diagram showing an example of the configuration of a vehicle-mounted camera device 10 that processes image information from multiple cameras. In the vehicle-mounted camera device 10, object recognition information is obtained using parallax information obtained from images of multiple cameras and is output to a vehicle control device side (not shown). In addition, multiple cameras C are examples and are not limited to the configuration described in this configuration diagram.

[0078] The images captured by the plurality of cameras C are sequentially input to the camera image acquisition unit 11 in the vehicle-mounted camera device 10 in a time series. These images are processed in the calibration execution unit 12 to correct the images by obtaining internal parameters such as lens focal length, external parameters indicating the position and posture of the camera, and distortion coefficients of the lens. In addition, the camera self-diagnosis unit 13 uses the images to perform camera diagnosis.

[0079] In addition, the camera parameter acquisition unit 18 acquires parameters of the multiple cameras mounted on the vehicle in advance. The parameters include, for example, the resolution of the imaging element, the pixel size, the pixel pitch, the focal length of the lens, the distortion parameter of the lens, the camera setting angle, etc. In the stereo matching processing unit 14, the parallax is calculated by searching for corresponding points in two images captured from different perspectives at the same time. In this way, the distance information to the objects (objects) that appear simultaneously in the two images that capture the common part can be obtained.

[0080] The stereo matching priority determination unit 15 is a function added in the present invention, and determines which combination should be prioritized when there are multiple combinations of images that can provide stereoscopic vision for multiple screens from multiple cameras.

[0081] In the parallax information integration unit 16, the parallax information generated by each camera pair is given priority according to the result of the stereo matching priority determination unit 15. Then, only the parallax information with a predetermined priority or higher is retained and integrated, and the distance to all or part of the objects in the captured image is obtained. In addition, in the object recognition unit 17, it is recognized whether the captured object is an object to be monitored or an object not to be monitored during vehicle travel, and the information of the object to be considered as the object and the distance to the object is output to the external vehicle control device side (not shown).

[0082] Fig.18 This is a configuration example in which recognition performance and robustness are prioritized, and a stereo matching priority determination unit 15 is configured after the stereo matching processing unit 14. After the stereo matching processing unit 14 obtains disparity information for a plurality of stereo regions, the stereo matching priority determination unit 15 determines the priority for these stereo regions and outputs it to the disparity information integration unit or object recognition unit at the subsequent stage. Thus, the priority when performing stereo matching processing will be applied to the images after the next acquired image.

[0083] In contrast, Fig.19 This shows an example of a configuration that is used when reducing the processing load is a priority. Fig.18 The difference of the illustrated configuration example is that the stereo matching processing unit 14 is arranged after the stereo matching priority determination unit 15. This allows the processing to be limited to a stereo region with a high priority, and the processing load can be reduced.

[0084] Next, the idea of ​​priority in the stereo matching priority determination unit 15 will be described. Here, when stereo areas overlap, priority is assigned to disparity information according to the following conditions. Priority is determined based on what is important (priority criteria), and multiple ideas are described here. Fig. 20 An example of a priority benchmark is shown in .

[0085] Priority criterion 1 is the idea of ​​giving priority to the parallax information of a camera pair with high angular resolution (narrow field of view, high resolution of the imaging element). Priority criterion 2 is the idea of ​​giving priority to the parallax information of a camera pair with a small angular difference in the direction of the optical axis of the camera. Priority criterion 3 is the idea of ​​giving priority to the parallax information of a region close to the center of the lens (regions with little influence of distortion and chromatic aberration). Priority criterion 4 is the idea of ​​giving priority to the parallax information of a camera pair with high calibration accuracy (reliability) or a camera pair with high stereo matching accuracy (less noise). Priority criterion 5 is the idea of ​​giving priority to the parallax information obtained by a camera pair with a short baseline length within a range close to the own vehicle, and giving priority to the parallax information obtained by a camera pair with a long baseline length within a range far from the own vehicle. Priority criterion 6 is the idea of ​​giving priority to the parallax information of a camera with less lens dirt, a camera with less influence of backlight (strong light sources such as sunlight, headlights, and street lights), and a camera with no malfunction. Furthermore, criteria other than these may also be used.

[0086] use Fig.21 The priority criterion 1 is described in detail. In this example, the three stereoscopic areas REA, REB, and REX are formed by the front cameras CFR and CFL and the side cameras CS. In this case, REX is a repeated stereoscopic area, and parallax information can be obtained from both the stereoscopic area formed by CFR and CFL and the stereoscopic area formed by CFL and CS, so it is studied from which area the parallax information should be obtained.

[0087] First, regarding the stereo area REA, this area is the stereo area of ​​the camera CFL with 8MP (Mega Pixel) and 120-degree field of view and the camera CFR with 8MP and 120-degree field of view. The angular resolution of the camera CFL = 8 / 120 [Mega pixel / deg], and the angular resolution of the camera CFR = 8 / 120 [Mega pixel / deg]. Therefore, the angular resolution of the stereo area REA is 8 / 120 = 67 [Mega pixel / deg].

[0088] Next, for the stereo area REB, this area is the stereo area of ​​the 8MP (Mega Pixel) camera CFL with a field of view of 120 degrees and the 8MP camera CS with a field of view of 150 degrees. The angular resolution of the camera CFL = 8 / 120 [Megapixel / deg], and the angular resolution of the camera CS = 8 / 150 [Mega pixel / deg]. Therefore, the angular resolution of the stereo area REB is 8 / 150 = 53 [Mega pixel / deg].

[0089] Therefore, the angular resolution of the stereoscopic area REA is greater than that of the stereoscopic area REB. At this time, according to the idea of ​​giving priority to the parallax information of the camera pair with high angular resolution (narrow field angle, high resolution of the imaging element), the parallax information of the repeated stereoscopic area REX gives priority to the stereoscopic area REA. The so-called "priority" means that, for example, only the parallax information obtained from the stereoscopic area REA can be used in the repeated stereoscopic area REX, or the parallax information obtained from the stereoscopic area REA can be weighted and used together with the parallax information obtained from the stereoscopic area REB.

[0090] use Fig. 22 The priority criterion 2 will be described in detail. In this example, three side cameras CS1, CS2 and rear camera CB form stereoscopic areas REA and REB. The stereoscopic areas REA and REB overlap in area REX, so area REX becomes an overlapping stereoscopic area.

[0091] First, for the stereo area REA, this area is the stereo area of ​​the 8MP (Mega Pixel) camera CS1 with a field of view of 150 degrees and the 8MP camera CS2 with a field of view of 150 degrees. The angular resolution of camera CS1 = 8 / 150 [Megapixel / deg], and the angular resolution of camera CS2 = 8 / 150 [Mega pixel / deg]. Therefore, the angular resolution of the stereo area REA is 8 / 150 [Mega pixel / deg], and the optical axis angle difference is 30 degrees.

[0092] Next, for the stereo area REB, this area is the stereo area of ​​the 5MP (Mega Pixel) camera CB with a field of view of 90 degrees and the 8MP camera CS2 with a field of view of 150 degrees. The angular resolution of the camera CR = 5 / 90 [Mega pixel / deg], and the angular resolution of the camera CS2 = 8 / 150 [Mega pixel / deg]. Therefore, the angular resolution of the stereo area REB is 8 / 150 [Mega pixel / deg], and the optical axis angle difference is 60 degrees.

[0093] In contrast, the stereoscopic area REX is an overlapping area of ​​the stereoscopic area REA and the stereoscopic area REB. Furthermore, the angular resolution of the stereoscopic area REA is the same as the angular resolution of the stereoscopic area REB. In addition, the optical axis angle difference of the stereoscopic area REA is smaller than the optical axis angle difference of the stereoscopic area REB. At this time, according to the idea of ​​giving priority to the parallax information of the camera pair with a small angle difference in the optical axis direction of the camera, the parallax information of the stereoscopic area REX gives priority to the stereoscopic area REA.

[0094] use Fig.23The priority criterion 3 is described in detail. In this example, three stereoscopic areas REA, REB, and REX are formed by the side cameras CSR and CSL and the rear camera CB. The stereoscopic areas REA and REB overlap in the area REX, so the area REX becomes an overlapping stereoscopic area. The stereoscopic area REX is further divided into REX1 and REX2.

[0095] First, for the stereo area REA, this area is the stereo area of ​​the 8MP (Mega Pixel) camera CSL with a field of view of 150 degrees and the 5MP camera CR with a field of view of 90 degrees. The angular resolution of the camera CSL = 8 / 150 [Megapixel / deg], and the angular resolution of the camera CB = 5 / 90 [Mega pixel / deg]. Therefore, the angular resolution of the stereo area REA is 8 / 150 [Mega pixel / deg], and the optical axis angle difference is 60 degrees.

[0096] Next, for the stereo area REB, this area is the stereo area of ​​the 8MP (Mega Pixel) camera CSR with a field of view of 150 degrees and the 5MP camera CB with a field of view of 90 degrees. The angular resolution of the camera CSR = 8 / 150 [Megapixel / deg], and the angular resolution of the camera CR = 5 / 90 [Mega pixel / deg]. Therefore, the angular resolution of the stereo area REB is 8 / 150 [Mega pixel / deg], and the optical axis angle difference is 60 degrees.

[0097] In contrast, the stereoscopic area REX is an overlapping area of ​​the stereoscopic area REA and the stereoscopic area REB. Furthermore, the angular resolution of the stereoscopic area REA is the same as the angular resolution of the stereoscopic area REB. In addition, the optical axis angle difference of the stereoscopic area REA is the same as the optical axis angle difference of the stereoscopic area REB. In addition, when the stereoscopic area REX is decomposed into REX1 and REX2, the angle difference between the center of REX1 and the center of the camera CSL is smaller than the angle difference between the center of REX2 and the center of the camera CSL, and the angle difference between the center of REX1 and the center of the camera CSR is larger than the angle difference between the center of REX2 and the center of the camera CSR. At this time, according to the idea of ​​giving priority to the parallax information close to the lens center (the area with less influence of distortion and chromatic aberration), the parallax information of the repeated stereoscopic area REX1 is given priority to the stereoscopic area REA, and the parallax information of the repeated stereoscopic area REX2 is given priority to the stereoscopic area REB.

[0098] use Fig.24The priority criterion 4 will be described in detail. In this example, three front cameras CFR, CFL and front camera CF form stereoscopic areas REA and REB. Since the stereoscopic areas REA and REB overlap in the area REX, the area REX becomes an overlapping stereoscopic area.

[0099] First, for the stereo area REA, this area is the stereo area of ​​the camera CF with an 8MP (Mega Pixel) and a 30-degree field of view and the camera CFL with an 8MP and a 120-degree field of view. The angular resolution of the camera CF = 8 / 30 [Mega pixel / deg], and the angular resolution of the camera CFL = 8 / 120 [Mega pixel / deg]. Therefore, the angular resolution of the stereo area REA is 8 / 120 [Mega pixel / deg], and the optical axis angle difference is 0 degrees. Furthermore, the camera CFR can also be used instead of the camera CFL.

[0100] Next, for the stereo area REB, this area is the stereo area of ​​the camera CFL with 8MP (Mega Pixel) and 120-degree field of view and the camera CFR with 8MP and 120-degree field of view. The angular resolution of the camera CFL = 8 / 120 [Megapixel / deg], and the angular resolution of the camera CFR = 8 / 120 [Mega pixel / deg]. Therefore, the angular resolution of the stereo area REB is 8 / 120 [Mega pixel / deg], and the optical axis angle difference is 0 degrees.

[0101] Therefore, the angular resolution of the stereo area REA is the same as the angular resolution of the stereo area REB. In addition, the angular difference of the optical axis of the stereo area REA is the same as the angular difference of the optical axis of the stereo area REB. In addition, the optical axes of the cameras CF, CFR and CFL are in the same direction, so the angular difference between the repeated stereo area REX and the center of the cameras CF, CFR and CFL is equal. Here, the angular resolution of the camera CF is high, and it is highly likely that a high-precision calibration can be performed using the white line in the distance. In other words, the calibration accuracy of the stereo area REA is higher than that of the stereo area REB. At this time, according to the idea of ​​giving priority to the parallax information of camera pairs with high calibration accuracy (reliability) and camera pairs with high stereo matching accuracy (less noise), the parallax information of the repeated stereo area REX gives priority to the stereo area REA.

[0102] use Fig.25The priority criterion 5 is described in detail. In this example, three stereoscopic areas REA and REB are formed by the front cameras CFR, CFL and the front camera CF. The stereoscopic areas REA and REB overlap in the area REX, so the area REX becomes an overlapping stereoscopic area. In addition, the overlapping stereoscopic area REX is divided into an area REXN that exists at a short distance from the own vehicle and an area REXF that exists at a long distance from the own vehicle.

[0103] First, regarding the stereo area REA, this area is the stereo area of ​​the camera CF with an 8MP (Mega Pixcel) field of view of 30 degrees and the camera CFL with an 8MP (Mega Pixcel) field of view of 120 degrees. The baseline length between the camera CF and the camera CFL is 15 cm. The so-called baseline length refers to the distance between the two monocular cameras that form the stereo camera. In addition, the camera CFR can also be used instead of the camera CFL.

[0104] Next, regarding the stereoscopic area REB, this area is the stereoscopic area of ​​the camera CFL of 8MP (Mega Pixcel) and the viewing angle of 120 degrees and the camera CFR of 8MP (Mega Pixcel) and the viewing angle of 120 degrees. The base line length between the camera CFL and the camera CFR is 30 cm.

[0105] Generally, stereo cameras with long baseline lengths (e.g., about 1m) have high distance resolution and small errors at long distances (e.g., more than 100m). On the other hand, at close distances (e.g., less than 10m), sometimes the object cannot be captured together, that is, the object is not included in the stereo area. In addition, the visual representation of the object varies greatly between cameras, so stereo matching is difficult. Therefore, stereo cameras with long baseline lengths have the property that long-distance distance measurement is better than short-distance distance measurement.

[0106] In contrast, stereo cameras with short baseline lengths (e.g., about 10 cm) can obtain small parallax at long distances (e.g., more than 100 m) and low distance resolution, so the distance measurement error is large. On the other hand, at close distances (e.g., less than 10 m), the object can be captured together, and its visual expression is similar between cameras, so stereo matching can be achieved. Therefore, stereo cameras with short baseline lengths have the property that short-distance distance measurement is better than long-distance distance measurement.

[0107] Therefore, in the range close to the own vehicle, the disparity information of the area REXN existing in the repeated stereo area REX at a distance close to the own vehicle is prioritized over the stereo area REA, according to the idea of ​​giving priority to the disparity information obtained by the camera pair with a short baseline length. In addition, in the range far from the own vehicle, the disparity information of the area REXF existing in the repeated stereo area REX at a distance far from the own vehicle is prioritized over the stereo area REB. Furthermore, the long distance or the short distance is not limited to the exemplified distance, and the priority criterion 5 can be set according to any distance.

[0108] use Fig.26 The priority criterion 6 will be described in detail. In this example, three side cameras CS1, CS2 and a rear camera CB form stereoscopic areas REA and REB. The stereoscopic areas REA and REB overlap in the area REX, so the area REX becomes an overlapping stereoscopic area.

[0109] First, regarding the stereo area REA, this area is the stereo area of ​​the camera CS1 with 8MP (Mega Pixel) and a field of view of 150 degrees and the camera CS2 with 8MP and a field of view of 150 degrees. The angular resolution of the camera CS1 = 8 / 150 [Megapixel / deg], and the angular resolution of the camera CS2 = 8 / 150 [Mega pixel / deg]. Therefore, the angular resolution of the stereo area REA is 8 / 150 [Mega pixel / deg], and the optical axis angle difference is 30 degrees.

[0110] For the stereo area REB, this area is the stereo area of ​​the 5MP (Mega Pixel) camera CB with a field of view of 90 degrees and the 8MP camera CS2 with a field of view of 150 degrees. The angular resolution of the camera CB = 5 / 90 [Mega pixel / deg], and the angular resolution of the camera CS2 = 8 / 150 [Mega pixel / deg]. Therefore, the angular resolution of the stereo area REB is 8 / 150 [Mega pixel / deg], and the optical axis angle difference is 60 degrees.

[0111] Therefore, the angular resolution of the stereoscopic area REA is the same as the angular resolution of the stereoscopic area REB. In addition, the optical axis angle difference of the stereoscopic area REA is smaller than the optical axis angle difference of the stereoscopic area REB. Here, because dirt is detected on the camera CS1, the visibility is deteriorated, and the visibility of the camera CS1 is not as good as the visibility of the camera CS2 and the visibility of the camera CB. At this time, according to the idea of ​​giving priority to the parallax information of the camera with less lens dirt and the camera without any malfunction, the parallax information of the repeated stereoscopic area REX gives priority to the stereoscopic area REB.

[0112] The priority criteria are mainly the above criteria, but in addition, it is advisable to consider Fig. 27 In this example, an overlapping stereoscopic area REX is formed by three cameras, namely, a narrow-angle front camera CF and left and right front cameras CFR and CFL, which are front cameras of the vehicle.

[0113] First, the stereo area REA is a stereo area captured by a stereo camera with a long horizontal baseline (a pair of cameras CFR and CFL), and the parallax accuracy of the vertical edge is high. On the other hand, the stereo area REB is a stereo area captured by a stereo camera with a long vertical baseline (a pair of camera CF and camera CFR or CFL), and the parallax accuracy of the horizontal edge is high.

[0114] Therefore, with respect to the disparity information of the repeated stereoscopic area REX, it is preferable to integrate the longitudinal edge disparity of the stereoscopic area REA and the lateral edge disparity of the stereoscopic area REB to generate dense disparity information. That is, in the case where stereoscopic areas with different baseline length directions are repeated, dense disparity information can be generated by integrating the disparity information. In addition, in the integration of disparity information, it is also possible to integrate after weighting a certain disparity information.

[0115] Fig.28 To express Fig.18 , Fig.19 Here, a priority is set for each stereo region that generates an overlapping stereo region. Fig. 20 The priority criteria shown have a weight W ( Fig.28 In the example, W1 to W7), each time a condition is met, the weight set for that condition is added. Finally, the combination with the highest weight is selected.

[0116] exist Fig.28 In the process of , all the three-dimensional areas that generate repeated three-dimensional areas are input, and they are selected in order for the subsequent judgment and weight addition processing. In addition, as a prerequisite for the subsequent processing, in the processing step S1, a weight W is set for each judgment item of the priority standard when it is consistent with it. What value the weight is set to can be set as appropriate.

[0117] In processing step S2, it is checked whether the difference in angular resolution between the stereoscopic areas is less than a predetermined value. If the difference in angular resolution is not less than the predetermined value, the priority of the stereoscopic area with high angular resolution is increased (weight W1 is added) in processing step S3.

[0118] In processing step S4, it is confirmed whether the angle difference in the optical axis direction between the two cameras forming each stereoscopic area is below the specified value. If the angle difference in the optical axis direction is not below the specified value, the priority of the stereoscopic area with a small angle difference in the optical axis direction is increased in processing step S5 (adding weight W2).

[0119] In processing step S6, it is confirmed whether the angle difference between the center of each stereoscopic area and the lens center of at least one camera in the camera pair forming the stereoscopic area is less than a specified value. If the angle difference with the lens center is not less than the specified value, the priority of the stereoscopic area close to the lens center is increased in processing step S7 (adding weight W3).

[0120] In processing step S8, it is checked whether the difference in calibration accuracy of each stereoscopic area is less than a specified value. If the difference in calibration accuracy is not less than the specified value, the priority of the stereoscopic area with high calibration accuracy is increased (weight W4 is added) in processing step S9.

[0121] In processing step S10, it is confirmed whether the difference in the baseline length of the camera pair forming each stereoscopic area is less than a specified value. If the difference in the baseline length is not less than the specified value, the process proceeds to processing step S11. In processing step S11, when the overlapping stereoscopic area is less than the specified measurement distance, the priority of the stereoscopic area formed by the camera pair with a short baseline length is increased, and when the overlapping stereoscopic area is greater than the specified measurement distance, the priority of the stereoscopic area formed by the camera pair with a long baseline length is increased (weight W5 is added).

[0122] In processing step S12, it is checked whether the difference in stereo matching accuracy between the three-dimensional areas is less than a predetermined value. If the difference in stereo matching accuracy is not less than the predetermined value, the priority of the three-dimensional area with high stereo matching accuracy is increased (weight W6 is added) in processing step S13.

[0123] In processing step S14, it is checked whether the difference in visibility of the camera pairs forming each stereoscopic area is below a specified value. If the difference in visibility is not below the specified value, the priority of the stereoscopic area with good camera visibility is increased in processing step S15 (weight W7 is added).

[0124] In the processing step S16, all conditions are confirmed for all three-dimensional areas that generate repeated three-dimensional areas, the total of the given weights is calculated, and the information of the three-dimensional area with a high priority P is output. It is also possible to output a result obtained by integrating all or part of the information of the three-dimensional area with a high priority P and the information of the three-dimensional area with a low priority P according to the priority P. In addition, the process is not necessarily limited to executing all the processing steps, and the order of each processing step can be changed.

[0125] Furthermore, the allocation amount of the computing resources of the stereo matching processing unit 15 and the object recognition unit 17 can be changed according to the priority P, or the priority P can be set according to the driving direction and driving environment of the vehicle 1. For example, when the vehicle 1 is driving in reverse, the priority P of the rear stereo area is set higher than the priority P of other areas, or when the vehicle 1 is driving in an area with a high risk of falling objects in the distance, the priority P of the distant area is set higher than the priority P of the nearby area. In this way, it is possible to appropriately set the important area whose priority should be increased according to the driving scene and improve the accuracy of the parallax information of the important area.

[0126] According to the third embodiment, a combination that can obtain high-precision parallax information can be determined for a stereoscopic region that generates an overlapping stereoscopic region, thereby ensuring high precision. Example 4

[0127] In Embodiment 4, a variation of the camera configuration that can realize a 360-degree multi-stereoscopic camera system is described. Although the description of the details is omitted, configurations corresponding to the respective purposes can be used.

[0128] Fig.29 Yes Figure 6 , Figure 7 , Fig.10 The camera group A has an additional configuration example of 4 cameras in the front, back, left and right, and the configuration of the camera group A has been appropriately changed.

[0129] Fig.30 Yes Figure 8 , Fig. 9 The camera group A1 has an additional configuration example of four cameras in the front, back, left and right directions, and the configuration of the camera group A1 has been appropriately changed.

[0130] Fig.31 This is an example of a configuration in which four cameras are added to the camera group A2 in the front, back, left and right directions. The configuration of the camera group A2 is appropriately changed.

[0131] Fig.32The figure shows the arrangement of four cameras on the vehicle in a configuration example in which four cameras are added to the camera group A2. Here, the camera group A2 ensures the field of view for long- to medium-distance, and the four additional cameras cover the short distance.

[0132] It should be noted that although these multiple cameras are installed on the periphery of the vehicle, they can be installed at any position inside or outside the outer contour of the vehicle as appropriate. Among them, the left and right cameras are preferably installed on the upper part of the ceiling in order to reduce the adhesion of dirt raised during driving or to conveniently shoot the distant from the highest possible angle.

[0133] Fig.33 This is a diagram showing the arrangement of four cameras on the vehicle in a configuration example in which four cameras are added to the camera group A. Here, the camera group A ensures a field of view for long to medium distances, and the four additional cameras cover short distances.

[0134] Fig.34 : is a diagram showing the arrangement of the four-sided cameras of the vehicle in the camera group A2. Here, the camera group A2 ensures a field of view area for long distance to medium distance.

[0135] Fig.35 : is a diagram showing the arrangement of four cameras on the vehicle in the camera group A1. Here, the camera group A1 ensures a field of view area for long distance to medium distance.

[0136] In addition, the present invention includes various variations and is not limited to the above-mentioned embodiments. For example, the above-mentioned embodiments are detailed descriptions for explaining the present invention in an easy-to-understand manner, and are not necessarily limited to all the structures described. In addition, a part of the structure of a certain embodiment can be replaced with the structure of other embodiments, and the structure of other embodiments can be added to the structure of a certain embodiment. In addition, other structures can be added, deleted, or replaced to a part of the structure of each embodiment. Explanation of symbols

[0137] 1…vehicle, 10…vehicle-mounted camera device, 11…camera image acquisition unit, C…camera, 12…calibration execution unit, 13…camera self-diagnosis unit, 14…stereo matching processing unit, 15…stereo matching priority determination unit, 16…parallax information integration unit, 17…object recognition unit, 18…camera parameter acquisition unit.

Claims

1. A camera system, characterized in that: have: A first camera group, which includes a plurality of cameras, wherein the plurality of cameras are configured to capture the entire surrounding environment of the own vehicle continuously in at least one direction; A second camera group, comprising a plurality of cameras, the plurality of cameras being configured to capture the entire surrounding environment around the own vehicle continuously at least in one direction, and being configured to have a camera field of view, i.e., a stereoscopic area, which at least overlaps with the camera field of view of the first camera group; as well as The stereo matching processing unit performs stereo matching processing using the images captured by the first camera group and the images captured by the second camera group to obtain parallax information.

2. The camera system according to claim 1, characterized in that The first camera group and the second camera group include at least three cameras configured to have a plurality of the three-dimensional areas. Ends of the plurality of three-dimensional regions contact or intersect with each other.

3. The camera system according to claim 1, characterized in that The first camera group and the second camera group include at least three cameras configured to have a plurality of the three-dimensional areas, and are configured to be able to capture the entire surrounding environment continuous in at least one direction within the three-dimensional area around the own vehicle.

4. The camera system according to claim 2 or 3, characterized in that The first camera group includes three monocular cameras for photographing the front, left rear, and right rear of the vehicle respectively. The second camera group includes three monocular cameras for photographing the left front, right front, and rear of the own vehicle respectively.

5. The camera system according to claim 2 or 3, characterized in that The first camera group includes three monocular cameras for photographing the front, left rear, and right rear of the vehicle respectively. The second camera group includes four monocular cameras for photographing the front, left front, right front, and rear of the own vehicle respectively.

6. The camera system according to claim 2 or 3, characterized in that The first camera group includes three monocular cameras for photographing the front, left rear, and right rear of the vehicle respectively. The second camera group includes four monocular cameras for photographing the front, left rear, right rear, and rear of the own vehicle respectively.

7. The camera system according to claim 1, characterized in that The first camera group and the second camera group include a plurality of monocular cameras.

8. The camera system according to claim 1, characterized in that The second camera group includes a plurality of cameras whose optical axis directions are different from those of the plurality of cameras of the first camera group.

9. The camera system according to claim 1, characterized in that The first camera group and the second camera group include a plurality of cameras with different optical axis directions.

10. The camera system according to claim 1, characterized in that A third camera group is provided at the front, rear, left and right sides of the vehicle.

11. The camera system according to claim 10, characterized in that The third camera group is a fisheye camera.

12. The camera system according to claim 10, characterized in that The third camera group is configured to have a camera field of view, that is, a stereoscopic area, which overlaps with the camera field of view of at least one of the first camera group or the second camera group. The stereo matching processing unit performs stereo matching processing using at least one of the images captured by the first camera group or the images captured by the second camera group and the images captured by the third camera group.

13. The camera system according to claim 1, characterized in that Among the cameras included in the first camera group and the second camera group, at least the cameras for photographing the left side and the right side of the own vehicle are arranged on the upper part of the ceiling of the own vehicle.

14. The camera system according to claim 1, characterized in that The cameras included in the first camera group and the second camera group are installed on the peripheral portion of the own vehicle.

15. The camera system according to claim 1, characterized in that The cameras included in the first camera group and the second camera group are arranged in a manner to form a repeated three-dimensional area in which a plurality of the three-dimensional areas are repeated.

16. The camera system according to claim 15, characterized in that The repeated three-dimensional area is formed in front of the vehicle or in the rear of the vehicle.

17. The camera system according to claim 15, characterized in that The method further includes a stereo matching priority determination unit configured to determine from which three-dimensional region among a plurality of three-dimensional regions forming the overlapping three-dimensional region the information acquired is given priority.

18. The camera system according to claim 17, characterized in that The stereo matching processing unit obtains disparity information of the stereo area determined to have a high priority by the stereo matching priority determination unit.

19. The camera system according to claim 17, characterized in that The stereo matching priority determination unit determines a priority for the plurality of stereoscopic areas for which parallax information has been obtained by the stereo matching processing unit.

20. The camera system according to claim 17, characterized in that The stereo matching priority determination unit determines that priority of parallax information of a stereoscopic area formed by a camera pair having a higher angular resolution among a plurality of stereoscopic areas forming the overlapping stereoscopic area is higher.

21. The camera system according to claim 17, characterized in that The stereo matching priority determination unit determines that the priority of parallax information of a stereoscopic area formed by a pair of cameras having a smaller angular difference in optical axis directions of the cameras among the plurality of stereoscopic areas forming the overlapping stereoscopic area is higher.

22. The camera system according to claim 17, characterized in that The stereo matching priority determination unit determines that the priority of parallax information from a stereoscopic area formed by a camera pair closer to a lens center among a plurality of stereoscopic areas forming the overlapping stereoscopic area is high.

23. The camera system according to claim 17, characterized in that The stereo matching priority determination unit determines that priority is high for parallax information from a stereoscopic region formed by a camera pair having high calibration accuracy among a plurality of stereoscopic regions forming the overlapping stereoscopic region.

24. The camera system according to claim 17, characterized in that The stereo matching priority determination unit determines that priority of parallax information from a stereoscopic region formed by a camera pair having high stereo matching accuracy among a plurality of stereoscopic regions forming the overlapping stereoscopic region is high.

25. The camera system according to claim 17, characterized in that The stereo matching priority determination unit determines that, among a plurality of stereo regions forming the overlapping stereo region, parallax information from a stereo region formed by a camera pair having a short base line length when the overlapping stereo region is less than a predetermined measurement distance has a high priority.

26. The camera system according to claim 17, characterized in that The stereo matching priority determination unit determines that, among a plurality of stereo regions forming the overlapping stereo region, a stereo region formed by a camera pair having a long base line length when the overlapping stereo region is at a predetermined measurement distance or longer has a higher priority.

27. The camera system according to claim 17, characterized in that The stereo matching priority determination unit determines that priority of parallax information on a stereoscopic area formed by a camera having less lens dirt or a camera having less influence of backlight among a plurality of stereoscopic areas forming the overlapping stereoscopic area is high.

28. The camera system according to claim 17, characterized in that The stereo matching priority determination unit determines that the priority of parallax information of a stereoscopic area formed by a camera pair having no malfunction among a plurality of stereoscopic areas forming the overlapping stereoscopic area is high.

29. The camera system according to claim 17, characterized in that The device further includes a parallax information integration unit that integrates the parallax information obtained by the plurality of camera pairs when the three-dimensional areas formed by the plurality of camera pairs having different base line length directions overlap.

30. The camera system according to claim 17, characterized in that When the overlapping three-dimensional area is formed, the stereo matching priority determination unit determines which three-dimensional area among the plurality of three-dimensional areas the information acquired from is prioritized based on at least one of the behavior of the own vehicle and the driving environment.

Images