Imaging device and program

Abstract

To reduce the deviation of scanning timing occurring in left and right imaging units having different angles of view.SOLUTION: An imaging device includes a first imaging unit having a rolling shutter type imaging element, a second imaging unit having a rolling shutter type imaging element with a resolution different from that of the first imaging unit, and a delay instructing unit that delays the scanning start timing of either the first imaging unit or the second imaging unit by a delay time which is the deviation of the scanning completion timing of the first imaging unit and the second imaging unit in an arbitrary region in the first captured image captured by the first imaging unit and an arbitrary region in the second captured image captured by the second imaging unit.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to an imaging device and a program. [Background technology]

[0002] Conventionally, stereo cameras, which have two cameras arranged on the left and right, are mounted on vehicles such as automobiles and are used to measure distance information from the vehicle to an object. In addition to distance information, stereo cameras are also used to recognize various objects around the vehicle. Such stereo cameras contribute to improving safety during vehicle driving assistance.

[0003] Note that object recognition can be performed using only images captured by a single camera, even without distance information. Therefore, it has been considered to configure a stereo camera with cameras with different angles of view, namely a wide-angle camera and a narrow-angle camera. With such a stereo camera, object recognition can be performed over a wide range using the wide-angle camera, and the distance to the object can be measured in the area where the imaging ranges of the narrow-angle camera and the wide-angle camera overlap.

[0004] Patent Document 1 discloses a technique for calculating a vertical timing delay time for correcting the misalignment from the amount of misalignment between the trimming positions of the left and right images, and instructing a vertical timing adjuster to make the adjustment so that the vertical misalignment approaches 0 spatially. By making such adjustments, it is possible to obtain a high-quality stereoscopic image with no misalignment between the left and right cameras. Summary of the Invention [Problem to be solved by the invention]

[0005] However, in a conventional vehicle-mounted stereo camera equipped with cameras with different angles of view, when a rolling shutter system is used as the shutter system, there is a problem that the scanning timing of the left and right cameras of the vehicle-mounted stereo camera is misaligned.

[0006] More specifically, the rolling shutter method scans the pixels in the sensor in sequence. Therefore, if the left and right cameras have different angles of view (amount of pixels scanned), even if they start scanning at the same time, there will be a timing difference during the scan. If there is a timing difference during the scan, the same object will be captured at different times by the left and right cameras, causing errors in the distance information.

[0007] The present invention has been made in view of the above, and has an object to reduce the misalignment in scanning timing that occurs between left and right imaging units having different angles of view. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems and achieve the object, the present invention provides an imaging device including a first imaging section having an imaging element of a rolling shutter type, a second imaging section having an imaging element of a rolling shutter type having a resolution different from that of the first imaging section, and a delay instruction section that delays a scanning start timing for one of the first imaging section and the second imaging section by a delay time that is a difference in scanning completion timing between the first imaging section and the second imaging section in an arbitrary area in a first captured image captured by the first imaging section and an arbitrary area in a second captured image captured by the second imaging section. the arbitrary region in the first captured image and the arbitrary region in the second captured image are regions in which the same object exists, and the delay instruction unit delays the scanning start timing so that a difference in scanning time to the object approaches zero. It is characterized by: Effect of the Invention

[0009] According to the present invention, it is possible to reduce the difference in scanning timing that occurs between left and right imaging units having different angles of view. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic plan view showing a configuration of an imaging device according to a first embodiment. [Diagram 2] FIG. 2 is a schematic plan view showing the configuration of the first imaging section and the second imaging section. [Diagram 3] FIG. 3 is a block diagram showing the hardware configuration of a control system of the imaging apparatus. [Figure 4] FIG. 4 is a diagram showing an example of magnification adjustment and trimming in the correction unit. [Diagram 5] FIG. 5 is a diagram showing an example of distance measurement by the imaging device. [Figure 6] FIG. 6 is a diagram showing an example of the horizontal angle of view of the first and second imaging units when they are arranged so that the Z axis is the optical axis direction. [Figure 7] FIG. 7 is a diagram showing an example of the vertical angle of view of the first and second imaging units when they are arranged so that the Z axis is the optical axis direction. [Figure 8] FIG. 8 is a diagram illustrating an example of the difference in scanning timing between the imaging element of the first imaging unit and the imaging element of the second imaging unit. [Figure 9] FIG. 9 is a diagram showing an example of the difference in scanning timing between front and rear lines of a subject. [Figure 10] FIG. 10 is a diagram showing a case where the scanning timing for the subject is different between the first imaging unit and the second imaging unit. [Figure 11] FIG. 11 is a diagram showing a case where the difference in scanning timing for a subject between the first imaging unit and the second imaging unit is brought close to zero. [Figure 12] FIG. 12 is a diagram showing an example in which the scanning timing is delayed. [Figure 13] FIG. 13 is a block diagram illustrating a hardware configuration of a control system of the imaging apparatus according to the second embodiment. As shown in FIG. [Figure 14] FIG. 14 is a diagram showing an example of vanishing points. [Figure 15] FIG. 15 is a flowchart showing the flow of processing for reflecting the delay time from the vanishing point. [Figure 16] FIG. 16 is a diagram showing an example in which an imaging device according to the embodiment is provided in an object detection device for an automobile. [Figure 17] FIG. 17 is a diagram showing a schematic configuration of an object detection device for an automobile to which the imaging device according to the embodiment is applied. [Figure 18]FIG. 18 is a diagram showing an example of the configuration of an object detection device for an automobile to which the imaging device according to the embodiment is applied. [Figure 19] FIG. 19 is a top view of an aircraft equipped with an imaging device according to an embodiment. [Figure 20] FIG. 20 is a front view of an aircraft equipped with an imaging device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of an imaging device and a program will be described in detail with reference to the accompanying drawings.

[0012] (First embodiment) Fig. 1 is a schematic plan view showing the configuration of an imaging device 10 according to a first embodiment, and Fig. 2 is a schematic plan view showing the configuration of a first imaging section 1 and a second imaging section 2. The stereo imaging device 10 has the first imaging section 1 and the second imaging section 2 as a stereo camera that captures an image of a subject.

[0013] The first imaging unit 1 and the second imaging unit 2 are disposed with a predetermined interval of base length B (e.g., 60 mm) between them. The imaging device 10 can capture images of the same subject by the first imaging unit 1 and the second imaging unit 2 with synchronized scanning timing.

[0014] 2, the first imaging unit 1 and the second imaging unit 2 include optical lenses 8 (8a, 8b) and imaging elements 9 (9a, 9b). The first imaging unit 1 and the second imaging unit 2 are fixed together by a stay member 12 while being held with a base length B therebetween.

[0015] The optical lens 8a is, for example, a lens with a focal length f of 2.5 mm, and the optical lens 8b is, for example, a lens with a focal length f of 5 mm.

[0016] The first imaging unit 1 is a wide-angle camera, and the second imaging unit 2 is a narrow-angle camera. That is, the first imaging unit 1 and the second imaging unit 2 have different resolutions, which results in different angles of view, and different focal lengths.

[0017] The imaging element 9a of the first imaging unit 1 is a high-resolution CMOS image sensor of the rolling shutter type. The imaging element 9a has a pixel pitch pp of 3 μm. The rolling shutter type is a type in which pixels in the sensor are scanned in sequence.

[0018] The imaging element 9b of the second imaging unit 2 is a low-resolution CMOS image sensor of a rolling shutter type. The imaging element 9b has a pixel pitch pp of 3 μm.

[0019] Next, the control system will be described.

[0020] The imaging device 10 has a hardware configuration that utilizes a conventional computer, and includes a control device such as a CPU (Central Processing Unit), a storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and an external storage device such as a HDD that stores a control program.

[0021] The CPU of the imaging device 10 executes a control program to function as a storage unit 3, a correction unit 6, a parallax calculation unit 7, and a delay instruction unit 5 as shown in FIG.

[0022] The control program executed by the imaging device 10 of the present embodiment may be stored in a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the control program executed by the imaging device 10 of the present embodiment may be provided or distributed via a network such as the Internet.

[0023] Furthermore, the control program executed by the imaging device 10 of this embodiment may be provided by being pre-installed in a ROM or the like.

[0024] 3 is a block diagram showing a hardware configuration of a control system of the imaging device 10. As shown in FIG. 3, the imaging device 10 includes a storage unit 3, a correction unit 6, a parallax calculation unit 7, and a delay instruction unit 5.

[0025] The storage unit 3 stores the captured image 11, the corrected image 14, the parallax image 15, and the delay time 17.

[0026] Here, the captured image captured by the first imaging unit 1 is referred to as the "first captured image." Also, the captured image captured by the second imaging unit 2 is referred to as the "second captured image." Also, when there is no need to distinguish between the first captured image and the second captured image, they are simply referred to as "captured images." The captured image 11 is either the first captured image or the second captured image.

[0027] The correction unit 6 adjusts the magnification and trims the first captured image and the second captured image so that they appear to be images captured at the same focal length, and corrects the coordinate shift of the first captured image (second captured image) caused by the position shift of the first imaging unit 1 (second imaging unit 2).

[0028] As described above, the first imaging unit 1, which is a wide-angle camera, and the second imaging unit 2, which is a narrow-angle camera, have different resolutions, which results in different angles of view, and different focal lengths. Therefore, it is necessary to adjust the focal length in order to obtain the distance to the subject, which will be described later. The correction unit 6 adjusts the magnification and performs trimming in order to adjust the focal length.

[0029] Fig. 4 is a diagram showing an example of magnification adjustment and trimming in the correction unit 6. In Fig. 4, the captured image captured by the first imaging unit 1, which is a wide-angle camera, is referred to as a first captured image 11-1, and the captured image captured by the second imaging unit 2, which is a narrow-angle camera, is referred to as a second captured image 11-2.

[0030] 4, the focal length of the first imaging unit 1 is f1, and the focal length of the second imaging unit 2 is f2. It is known that the magnification can be calculated by the ratio of the focal lengths, and the magnification m of the second captured image 11-2 to the first captured image 11-1 is (f2 / f1). Therefore, the correction unit 6 enlarges the first captured image 11-1 by m times and trims it to the same sizes X1', Y1' as the sizes X2, Y2 of the second captured image 11-2, thereby creating a first corrected image 14-1 with the same magnification and image size as the second captured image 11-2.

[0031] In this embodiment, as described above, the focal length f1 of the optical lens 8a of the first imaging unit 1 is 2.5 mm, and the focal length f2 of the optical lens 8b of the second imaging unit 2 is 5 mm. In such a case, the focal length of the captured image is adjusted, for example, by adjusting the magnification and trimming, as follows: Apparent focal length of the image captured by the first imaging unit 1: 5 mm Apparent focal length of the image captured by the second imaging unit 2: 5 mm

[0032] The parallax calculation unit 7 calculates the parallax from the first corrected image obtained by correcting the first captured image by the correction unit 6 and the second captured image. Then, the parallax calculation unit 7 uses the calculated parallax to generate a parallax image 15 including information on the parallax of the coordinates of the captured image 11. A specific method of calculating the parallax is similar to that of a well-known parallel equilateral stereo camera.

[0033] The disparity calculation unit 7 calculates the correlation while shifting the image area around the feature point extracted from the reference captured image and the corresponding candidate area on the captured image to be compared by one pixel unit, and calculates the disparity with an accuracy of less than a pixel unit by interpolating the correlation value near the maximum correlation value using a parabola, etc. The disparity calculation unit 7 of this embodiment generates a disparity image 15 with an accuracy of 1 / 10 pixel or less.

[0034] Here, the principle of distance measurement in the imaging device 10 using the first imaging section 1 and the second imaging section 2 will be described.

[0035] Here, Fig. 5 is a diagram showing an example of distance measurement by the imaging device 10. As shown in Fig. 5, the first imaging unit 1 has a focal length f of the optical lens 8a, an optical center O0 of the optical lens 8a, and an imaging surface S0 of the imaging element 9a, and the Z axis is the optical axis direction. The second imaging unit 2 has a focal length f of the optical lens 8b, an optical center O1 of the optical lens 8b, and an imaging surface S1 of the imaging element 9b, and the Z axis is the optical axis direction.

[0036] The first imaging unit 1 and the second imaging unit 2 are disposed parallel to the X-axis and spaced apart by a base length B (for example, 60 mm).

[0037] The subject A is located at a distance d from the optical center O0 of the first imaging unit 1 in the optical axis direction.

[0038] The first imaging unit 1 forms an image of the subject A at P0, which is the intersection point of the straight line A-O0 and the imaging plane S0. Meanwhile, the second imaging unit 2 forms an image of the subject A at P1, which is the intersection point of the straight line A-O1 and the imaging plane S1.

[0039] Here, P0' is the intersection point between a line that passes through the optical center O1 of the second imaging unit 2 and is parallel to the line A-O0, and the imaging plane S1. Also, p is the distance between P0' and P1. The parallax calculation unit 7 calculates the distance p, which is the amount of positional shift on the images of the same subject A captured by the first imaging unit 1 and the second imaging unit 2, as the parallax. Then, the parallax calculation unit 7 uses the calculated parallax to generate a parallax image 15 including information on the parallax of the coordinates of the captured image 11.

[0040] 5, the triangle A-O0-O1 and the triangle O1-P0'-P1 are similar. Therefore, the distance d to the subject A can be calculated from the base line length B, the focal length f, and the parallax p as shown in the following formula. d=B×f / p

[0041] The imaging device 10 periodically and repeatedly performs imaging, correction, and parallax calculation of the captured image using the above-mentioned components (the first imaging unit 1, the second imaging unit 2, the memory unit 3, the correction unit 6, and the parallax calculation unit 7), thereby continuously generating a parallax image 15.

[0042] Incidentally, in the imaging device 10 equipped with the first imaging unit 1 and the second imaging unit 2 having different angles of view (amount of pixels scanned) as in this embodiment, when the rolling shutter method is adopted as the shutter method, even if scanning is started simultaneously, there is a problem that a timing difference occurs during scanning in the first imaging unit 1 and the second imaging unit 2. If a timing difference occurs during scanning in this way, the same subject will be imaged at different times on the left and right, causing an error in distance information.

[0043] Here, an example will be described in which an error occurs in distance information when a rolling shutter method is adopted as the shutter method in an imaging device 10 equipped with a first imaging unit 1 and a second imaging unit 2 having different angles of view.

[0044] 6 is a diagram showing an example of the horizontal angle of view of the first imaging unit 1 and the second imaging unit 2 when the Z axis is arranged as the optical axis direction. As shown in Fig. 6, when the Z axis is arranged as the optical axis direction, the horizontal angle of view of the first imaging unit 1, which is a wide-angle camera, is represented by reference symbol 1a, and the horizontal angle of view of the second imaging unit 2, which is a narrow-angle camera, is represented by reference symbol 2a.

[0045] 7 is a diagram showing an example of the vertical angle of view of the first imaging unit 1 and the second imaging unit 2 when the Z axis is arranged as the optical axis direction. As shown in Fig. 7, when the Z axis is arranged as the optical axis direction, the vertical angle of view of the first imaging unit 1, which is a wide-angle camera, is represented by reference symbol 1b, and the vertical angle of view of the second imaging unit 2, which is a narrow-angle camera, is represented by reference symbol 2b.

[0046] 8 is a diagram illustrating an example of the difference in scanning timing between the imaging element 9a of the first imaging unit 1 and the imaging element 9b of the second imaging unit 2. Fig. 8 illustrates the difference Δtv in scanning timing for subject A when the scanning timing of the imaging element 9a of the first imaging unit 1, which is a wide-angle camera, and the scanning timing of the imaging element 9b of the second imaging unit 2, which is a narrow-angle camera, are matched.

[0047] The rolling shutter type imaging devices 9 (9a, 9b) scan images line by line from the top to the bottom of the screen. As shown in FIG. 8, let the vertical line that scans up to the subject A in the imaging device 9a of the first imaging unit 1 be v1, and the vertical line that scans up to the subject A in the imaging device 9b of the second imaging unit 2 be v2. As shown in FIG. 8, the number of vertical lines that scan up to the subject A is such that v1 > v2.

[0048] Also, as shown in FIG. 8, let the scanning time per line in the imaging device 9a of the first imaging unit 1 be th1, and the scanning time per line in the imaging device 9b of the second imaging unit 2 be th2. Therefore, the scanning time tv1 up to the subject A in the imaging device 9a of the first imaging unit 1 is represented by (th1 × v1), and the scanning time tv2 up to the subject A in the imaging device 9b of the second imaging unit 2 is represented by (th2 × v2).

[0049] That is, as shown in FIG. 8, when the scanning time per line th1 in the imaging device 9a of the first imaging unit 1 is different from the scanning time per line th2 in the imaging device 9b of the second imaging unit 2, a time difference Δtv as shown in the following formula (1) occurs at the scanning timing of the subject A. Δtv = tv1 - tv2 = (th1 × v1) - (th2 × v2) ··(1)

[0050] Hereinafter, the case where th1 < th2 will be described as an example.

[0051] FIG. 9 is a diagram exemplarily showing the difference in scanning timing between the front and rear lines with respect to the subject A. FIG. 9 explains the difference in scanning timing between the front and rear lines with respect to the subject A when the scanning timings of the imaging device 9a of the first imaging unit 1, which is a wide-angle camera, and the imaging device 9b of the second imaging unit 2, which is a narrow-angle camera, are synchronized. As shown in FIG. 9, since the scanning time per line th1 in the imaging device 9a of the first imaging unit 1 and the scanning time per line th2 in the imaging device 9b of the second imaging unit 2 are in the relationship of th1 < th2, the time difference Δtv in the scanning timing of the subject A becomes smaller with each progress of the scanning.

[0052] Here, a problem that occurs when the scanning timing for the subject A differs between the first imaging unit 1 and the second imaging unit 2 will be described.

[0053] Fig. 10 is a diagram showing a case where the scanning timing for subject A is different between the first imaging unit 1 and the second imaging unit 2. Fig. 10 shows that, with the imaging surface S'0 of the first imaging unit 1 and the imaging surface S'1 of the second imaging unit 2, if the first imaging unit 1 images the subject A and the subject A moves a distance g to the left during a time Δth, the second imaging unit 2 images the subject A2 after the movement. As a result, as shown in Fig. 10, the parallax becomes large, and a distance d' to the position of the subject A' is obtained that is closer than the actual distance to the subject A.

[0054] As described above, the apparent focal length f of the optical lens 8 (8a, 8b) after trimming to adjust the focal length is 5 mm, the baseline length B which is the distance between the first imaging unit 1 and the second imaging unit 2 is 60 mm, and the pixel pitch pp is 3 μm.

[0055] In addition, the scanning time th1 per line of the first imaging unit 1 is 20 μs, the number of lines scanned to subject A is 480, the scanning time th2 per line of the second imaging unit 2 is 50 μs, and the number of lines scanned to subject A is 240.

[0056] The difference Δtv in scanning timing for subject A is −2.4 ms from the above formula (1).

[0057] Here, it is assumed that the distance d to subject A is 30 m, and subject A is moving to the left at a speed of 60 km / h.

[0058] The distance g that the subject A moves during the 2.4 ms from when the first imaging unit 1 scans the subject A until the second imaging unit 2 captures the image of the subject A is: g = 60km / h x 2.4ms = 0.04m It becomes.

[0059] Here, the increment in parallax caused by moving the distance g is defined as G. Furthermore, triangle A-A2-O1 and triangle O1-P1-P1' are similar. Therefore, the increment in parallax G is approximately 2.2 pixels according to the following formula. G=(((f×g) / d)) / pp

[0060] By the way, the parallax value p for a distance of 30 m to the original subject A is about 3.3 pixels, since the triangles A-O0-O1 and O1-P0'-P1 are similar, so P=((B×f) / d) / pp. Therefore, the parallax value p' is about 5.5 pixels.

[0061] 10, the triangle A'-O0-O1 and the triangle O1-P0'-P1 are similar, so d'=B×f / (p'×pp), and the distance d' to the subject A' is approximately 18 m.

[0062] In other words, the imaging device 10 measures the subject A to be 18 m away when it is actually 30 m away. If the braking distance of the vehicle is 20 m while the vehicle is moving, the object detection device of the vehicle equipped with the imaging device 10 will predict a collision based on this result alone because an object is present within the braking distance. In other words, the control system may determine that emergency braking or avoidance operations are necessary.

[0063] However, in general, an object detection device for an automobile has a margin, such as requiring multiple collision judgments before starting control. Therefore, although erroneous control does not necessarily occur in the above situation, it is natural that it is desirable for the stereo camera module to have high reliability on its own. In this way, the combination of the first imaging unit 1, which is a wide-angle camera, and the second imaging unit 2, which is a narrow-angle camera, can cause a problem in which the subject appears to be moving.

[0064] Therefore, in this embodiment, the imaging device 10 adjusts the scanning timing. Specifically, as shown in Fig. 3, the imaging device 10 includes a delay instruction unit 5. Also, as shown in Fig. 3, the imaging device 10 stores a delay time 17 in the storage unit 3.

[0065] The delay instruction unit 5, the details of which will be described later, reads out the delay time 17 from the storage unit 3, and instructs the imaging device 10 to delay the scan start timing of the first imaging unit 1 and the second imaging unit 2.

[0066] 11 is a diagram showing a case where the difference in scanning timing for subject A between the first imaging unit 1 and the second imaging unit 2 is brought closer to zero. In Fig. 11, a delay time Δtv in the scanning start timing is explained when the difference in scanning timing for subject A between the imaging element 9a of the first imaging unit 1, which is a wide-angle camera, and the imaging element 9b of the second imaging unit 2, which is a narrow-angle camera, is brought closer to zero.

[0067] As shown in FIG. 8, when scanning is started at the same scanning timing, the time until the subject A is scanned is as follows for the image sensor 9a of the first image capturing unit 1: tv1=th1×v1 In the image sensor 9b of the second image capturing unit 2, tv2=th2×v2 The difference Δtv in the scanning timing of the first imaging unit 1 and the second imaging unit 2 with respect to the subject A is given by Δtv=tv1-tv2=(th1×v1)-(th2×v2) It can be seen that.

[0068] Therefore, as shown in FIG. 11, by shifting the scan start timing of the image sensor 9a of the first imaging unit 1 and the image sensor 9b of the second imaging unit 2 by Δtv, the shift in scan timing of the subject A can be reduced.

[0069] That is, the imaging device 10 stores in the storage unit 3 as the delay time 17 the delay Δtv between the scanning timing of the first imaging unit 1 and the second imaging unit 2 with respect to the subject A.

[0070] Then, the delay instruction unit 5 of the imaging device 10 reads the delay time 17 from the storage unit 3 and instructs the delay of the scanning timings of the first imaging unit 1 and the second imaging unit 2.

[0071] FIG. 12 is a diagram showing an example in which the scanning timing is delayed. As shown in FIG. 12, based on the deviation Δtv of the scanning timing, which is the delay time 17 read from the storage unit 3 by the delay instruction unit 5 of the imaging device 10, the delay instruction unit 5 delays the scanning timing of the imaging element 9a of the first imaging unit 1, which is a wide-angle camera, and the scanning timing of the imaging element 9b of the second imaging unit 2, which is a narrow-angle camera, so that the difference therebetween approaches zero.

[0072] According to the example shown in FIG. 12, since the scanning time per line th1 in the imaging element 9a of the first imaging unit 1 and the scanning time per line th2 in the imaging element 9b of the second imaging unit 2 have the relationship of th1 < th2, the difference in the scanning timing increases with each advancement of the scanning. However, according to the example shown in FIG. 12, compared with the example shown in FIG. 9, the deviation of the scanning timing between the lines before and after the subject A is reduced.

[0073] Thus, according to the present embodiment, at the scanning start timing of the wide-angle camera (first imaging unit 1) and the narrow-angle camera (second imaging unit 2) having imaging elements with different resolutions, in advance, based on the difference in the scanning completion timings of the left and right imaging elements, one of the imaging elements is delayed by the deviation time of the scanning completion timing, and the scanning start trigger of the imaging element is changed so that the scanning timings of the left and right imaging elements in an arbitrary region (a region where the same subject exists) become the same. Thereby, the deviation of the scanning timing generated between the left and right imaging units with different imaging angles can be reduced.

[0074] Note that, in the above description, the case of the subject A has been described as an example, but it goes without saying that the target for reducing the scanning timing deviation may be adjusted according to an arbitrary region.

[0075] In the above description, the case where th1 < th2 has been described as an example. However, even when th1 > th2, a time difference of Δtv occurs in the scanning timing of the subject A, and by shifting the scanning start timing of the imaging element 9a of the first imaging unit 1 and the imaging element 9b of the second imaging unit 2 by Δtv, it is possible to reduce the deviation in the scanning timing of the subject A.

[0076] (Second Embodiment) Next, the second embodiment will be described. In the first embodiment, an area where the same subject exists is applied as an arbitrary area in the captured image. However, in the second embodiment, the difference from the first embodiment is that an area where a vanishing point exists where the direction in which a stationary object moves on the captured image converges is applied as an arbitrary area in the captured image. In the following description of the second embodiment, the description of the same parts as those in the first embodiment will be omitted, and the parts different from the first embodiment will be described.

[0077] FIG. 13 is a block diagram showing the hardware configuration of the control system of the imaging device 10 according to the second embodiment. As shown in FIG. 13, the imaging device 10 further includes a vanishing point detection unit 4 that performs detection of a vanishing point and the like with respect to the imaging device 10 shown in FIG. 3.

[0078] Here, the vanishing point will be described.

[0079] FIG. 14 is a diagram exemplarily showing a vanishing point. The example shown in FIG. 14 shows a case where the first imaging unit 1 which is a wide-angle camera and the second imaging unit 2 which is a narrow-angle camera are arranged with a depression angle with respect to the optical axis direction along the Z axis. In this case, the vertical field angle of the first imaging unit 1 which is a wide-angle camera is represented by reference numeral 1c, and the vertical field angle of the second imaging unit 2 which is a narrow-angle camera is represented by reference numeral 2c.

[0080] In a vehicle with a high vehicle height such as a truck, the camera mounting position becomes high, and since the reflection of the road surface on the near side is reduced, as shown in FIG. 14, it is conceivable to make the road surface reflect by attaching a depression angle.

[0081] Normally, collision determination in a vehicle control system is performed for objects in the traveling direction. For this reason, it is important to synchronize the scanning timing with respect to a subject A captured in the traveling direction D of the imaging device 10, as shown in FIG.

[0082] 14, subject A captured in the moving direction D of imaging device 10 is captured at the same position as the vanishing point where the directions of motion of stationary objects converge on the captured image. Therefore, imaging device 10 may be adapted to match the scanning timing with the vanishing point on the captured image.

[0083] The vanishing point detection unit 4 detects a vanishing point by reading out the first captured image 11-1 captured by the first imaging unit 1 and the second captured image 11-2 captured by the second imaging unit 2. Methods for detecting a vanishing point from captured images include, for example, the methods described in the following documents. References: Yamamoto Muneta and Kaneko Kunihiko (2011). Parallelization of vanishing point trajectory estimation processing from a moving camera image database using MapReduce, 3-4. https: / / db-event.jpn.org / deim2011 / proceedings / pdf / e9-2.pdf

[0084] Then, the vanishing point detection unit 4 writes, in the storage unit 3 as the delay time 17, the delay time at which the scanning timings at the vanishing points on the captured images of the first imaging unit 1 and the second imaging unit 2 coincide with each other.

[0085] In the above explanation, the vehicle travel direction is detected using the vanishing point on the captured image, but this is not limited to this, and the vehicle travel direction may be detected using something other than the vanishing point on the captured image.

[0086] Next, the process in which the vanishing point detection unit 4 reflects the delay time from the vanishing point will be described in detail.

[0087] Fig. 15 is a flowchart showing a process flow for reflecting a delay time from a vanishing point. As shown in Fig. 15, the vanishing point detection unit 4 reads out the first captured image 11-1 captured by the first imaging unit 1, and detects the vanishing point F1 (step S1).

[0088] Next, the vanishing point detection section 4 reads out the second captured image 11-2 captured by the second imaging section 2, and detects the vanishing point F2 (step S2).

[0089] The detection of the vanishing point F1 of the first imaging unit 1 and the detection of the vanishing point F2 of the second imaging unit 2 may be processed in parallel.

[0090] Next, the vanishing point detection unit 4 calculates a delay time Δtv of the scanning timing from the number of vertical lines v1 of the vanishing point F1 and the number of vertical lines v2 of the vanishing point F2 (step S3). Δtv=tv1-tv2=(th1×v1)-(th2×v2)

[0091] Then, the vanishing point detection unit 4 writes the delay time Δtv of the scanning timing into the storage unit 3 as the delay time 17 (step S4).

[0092] Thus, according to this embodiment, when starting scanning of the wide-angle camera (first imaging unit 1) and narrow-angle camera (second imaging unit 2) having imaging elements with different resolutions, one of the imaging elements is delayed in advance by the time difference in the scanning completion timing due to the difference in the scanning completion timing of the left and right imaging elements, and the scanning start trigger of the imaging element is changed so that the scanning timing of the left and right imaging elements is the same in any area (an area where a vanishing point exists where the directions of moving stationary objects converge). This makes it possible to reduce the difference in scanning timing that occurs between the left and right imaging units with different angles of view.

[0093] (Example) (Example of application to automobiles) Next, the imaging device 10 of each of the above embodiments can be provided in a moving object such as a robot device, an automobile, an aircraft, a ship, etc. An application example in which the imaging device 10 is applied to a moving object will be described below.

[0094] First, Fig. 16 is a diagram showing an example in which an imaging device 10 according to an embodiment is provided in an object detection device for an automobile. As shown in Fig. 16, the object detection device including the imaging device 10 is provided on the windshield of an automobile, etc., and captures images of pedestrians, road signs, the road surface, a preceding vehicle, guard rails, etc., within a predetermined imaging range ahead in the traveling direction. As described above, the imaging device 10 has a stereo camera configuration, and captures two images, one for the left eye and one for the right eye.

[0095] 17 is a diagram showing a schematic configuration of an object detection device. The object detection device outputs two images captured by an imaging device 10 to a vehicle ECU (Engine Control Unit) 50. The vehicle ECU 50 analyzes a parallax image based on the images from the imaging device 10, and performs control such as engine control, braking control, lane keeping assist, steering assist, etc. of the automobile based on the analysis results.

[0096] Fig. 18 is a diagram showing the configuration of an object detection device. As shown in Fig. 18, an imaging device 10 including a first imaging unit 1 and a second imaging unit 2 is connected to an image processing device 30. The first imaging unit 1 and the second imaging unit 2 include lenses 8a and 8b, imaging elements 9a and 9b, and sensor controllers 23a and 23b, respectively. The sensor controller 23 performs, for example, exposure control, image read control, communication with external circuits, and image data transmission control for each of the imaging elements 9a and 9b.

[0097] The image processing device 30 is provided, for example, in a vehicle ECU 50 shown in Fig. 17. The image processing device 30 includes, for example, a data bus line 300, a serial bus line 302, a CPU (Central Processing Unit) 304, an FPGA (Field-Programmable Gate Array) 306, a ROM (Read Only Memory) 308, a RAM (Random Access Memory) 310, a serial IF (Interface) 312, and a data IF (Interface) 314.

[0098] The imaging device 10 is connected to the image processing device 30 via a data bus line 300 and a serial bus line 302. A CPU 304 controls the operation of the entire image processing device 30 and executes image processing and image recognition processing. Images captured by the image sensors 9a, 9b of the first imaging unit 1 and the second imaging unit 2 are written to a RAM 310 of the image processing device 30 via the data bus line 300. Change control data for sensor exposure values, change control data for image read parameters, various setting data, and the like from the CPU 304 or FPGA 306 are transmitted and received via the serial bus line 302.

[0099] The FPGA 306 performs processes that require real-time performance on the image data stored in the RAM 310, such as gamma correction, distortion correction (parallelization of left and right images), and parallax calculation using block matching to generate the above-mentioned parallax image, and writes it again to the RAM 310. The parallax image is information that corresponds to the vertical position, horizontal position, and depth position of an object.

[0100] The CPU 304 controls the sensor controllers 23a and 23b of the imaging device 10 and the overall control of the image processing device 30. The CPU 304 acquires, for example, CAN (Controller Area Network) information of the host vehicle as parameters (vehicle speed, acceleration, steering angle, yaw rate, etc.) via the data IF 314.

[0101] Vehicle control data, which is recognition data of objects such as preceding vehicles, people, guardrails, and road surfaces, is supplied to the vehicle ECU 50 via the serial IF 312 and is used in, for example, an automatic braking system or a driving assist system provided as a control function of the vehicle ECU 50. The automatic braking system performs braking control of the vehicle. The driving assist system performs lane keeping assist and steering assist for the vehicle.

[0102] (Example of application to aircraft) Next, Fig. 19 and Fig. 20 are diagrams showing an example of installation of the imaging device 10 of the embodiment on an aircraft. Fig. 19 shows a top view of the aircraft provided with the imaging device of the embodiment, and Fig. 20 shows a front view of the aircraft provided with the imaging device of the embodiment. As shown in Fig. 19 and Fig. 20, the aircraft is a remotely controllable aircraft having a frame 101, a drive unit 102, rotors 103, and legs 104. The aircraft also has an imaging device 10 that captures still images or video of a subject.

[0103] As shown in Fig. 19, for example, the frame 101 is arranged such that the web portion 101a and the flange portions 101b arranged at both ends of the web portion 101a form an H-shape in a plan view. The shape of the frame 101 is not limited to this. For example, the frame 101 may have four arms extending radially from the center of the aircraft and arranged in an X-shape at right angles to each other. The frame 101 may also have a structure in which arms extend radially from the center in the same number as the driving devices 102 and rotors 103.

[0104] The driving device 102 is a motor that rotates the rotor 103. As an example, a brushless motor can be used as the driving device 102. The driving devices 102 are attached to the tips of the two flanges 101b, respectively, and are connected to the rotating shafts of the rotor 103. The driving devices 102 are controlled by the control device 100 independently of each other.

[0105] The rotors 103 are attached to the respective driving devices 102, and rotate when the driving devices 102 are driven. The rotation speed of the rotors 103 is adjusted by independently controlling the driving of each driving device 102. Although FIG. 19 illustrates the flying object having four rotors 103, the flying object can fly stably if it has three or more rotors 103. Furthermore, it is preferable to provide an even number of rotors 103, such as four, six, or eight. Moreover, the rotors 103 are not limited to a form in which a plurality of rotors 103 are arranged on approximately the same plane as shown in FIG. 19 and FIG. 20, but may be a form in which a plurality of rotors 103 arranged on approximately the same plane are stacked in multiple stages.

[0106] The legs 104 are connected to the underside of the frame 101 at approximately the center. The lower side of the legs 104 is formed in an inverted V shape to protect the aircraft from impacts when it lands on the ground.

[0107] The control device 100 is provided, for example, in a web portion 101a of a frame 101. The control device 100 controls each drive device 102 etc. in response to signals from various sensors provided in the aircraft according to the purpose and operation signals transmitted from the outside.

[0108] The control device 100 also independently controls the drive of each drive device 102, thereby independently controlling the rotation speed of each rotor 103, and controls the flight object's movements such as ascent, descent, forward movement, backward movement, rotation around the yaw axis (Z axis), and movement into the air for hovering (stationary flight). The control device 100 also controls the flight object's movement in the left and right directions, as well as forward and backward movement, by tilting the attitude of the flight object to the left and right.

[0109] The imaging device 10 is provided on the web portion 101a via a spacer that absorbs vibrations transmitted through the web portion 101a of the frame 101. This reduces the effect of vibrations of the frame 101 and the web portion 101a on the measurement of the distance to the structure. As will be described later, the support member (housing) of the imaging device 10 is formed of a carbon fiber reinforced plastic member (CFRP: Carbon Fiber Reinforced Plastic) with adjusted fiber direction.

[0110] This reduces the influence of external loads, such as bending, twisting, and stretching, that are applied to the imaging device 10 due to direct external factors such as wind and contact. Therefore, the imaging device 10 is able to measure the measurement target (object) with high accuracy by using the above-mentioned spacer and CFRP with adjusted fiber direction.

[0111] Such an imaging device 10 captures images of the surroundings of the flying object. As an example, the imaging device 10 is a so-called stereo camera device equipped with an imaging element 9. The imaging device 10 is fixed to a web portion 101a of a frame 101. The imaging device 10 is provided on the web portion 101a so that the optical axis is horizontal when the flying object flies horizontally parallel to the ground. The imaging device 10 is also provided on the web portion 101a so that the imaging direction coincides with the forward direction of the flying object.

[0112] Note that by providing a plurality of imaging devices 10, it may be possible to capture a full celestial sphere (360°) panoramic image around the flying object (full celestial sphere camera device).

[0113] In this way, the imaging device 10 of the embodiment can be installed in a moving object such as an automobile or an aircraft. It can also be applied to an industrial robot, etc. In either case, the same effects as described above can be obtained.

[0114] Such novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. Such embodiments and modifications are within the scope and spirit of the invention, and are included in the scope of the invention and its equivalents as set forth in the claims. [Explanation of symbols]

[0115] 1 First imaging unit 2 Second imaging section 4 Vanishing point detection section 5 Delay indicator 10. Imaging device [Prior art documents] [Patent documents]

[0116] [Patent Document 1] Patent No. 5683025

Claims

1. A first imaging unit having a rolling shutter type imaging element; a second imaging unit having a rolling shutter type imaging element having a different resolution from that of the first imaging unit; a delay instruction unit that delays a scanning start timing for one of the first imaging unit and the second imaging unit by a delay time that is a difference in scanning completion timing between the first imaging unit and the second imaging unit in an arbitrary region in a first captured image captured by the first imaging unit and an arbitrary region in a second captured image captured by the second imaging unit; Equipped with the arbitrary region in the first captured image and the arbitrary region in the second captured image are regions in which the same subject exists, the delay instruction unit delays the scanning start timing so that a difference in scanning time to the subject approaches zero.

1. An imaging device comprising:

2. A first imaging unit having a rolling shutter type imaging element; a second imaging unit having a rolling shutter type imaging element having a different resolution from that of the first imaging unit; a delay instruction unit that delays a scanning start timing for one of the first imaging unit and the second imaging unit by a delay time that is a difference in scanning completion timing between the first imaging unit and the second imaging unit in an arbitrary region in a first captured image captured by the first imaging unit and an arbitrary region in a second captured image captured by the second imaging unit; Equipped with the arbitrary region in the first captured image and the arbitrary region in the second captured image are regions where a vanishing point exists where directions of movement of a stationary object converge on the first captured image and the second captured image, the delay instruction unit delays the scanning start timing so that the scanning timings at the vanishing points coincide with each other.

1. An imaging device comprising:

3. a vanishing point detection unit that detects a vanishing point to which directions of movement of a stationary object converge on the first captured image and the second captured image; 3. The imaging device according to claim 2.

4. A first imaging unit having a rolling shutter type imaging element; a second imaging unit having a rolling shutter type imaging element having a different resolution from that of the first imaging unit; A computer that controls an imaging apparatus comprising: a delay instruction unit that delays a scanning start timing of one of the first imaging unit and the second imaging unit by a delay time that is a difference in scanning completion timing between the first imaging unit and the second imaging unit in an arbitrary region in a first captured image captured by the first imaging unit and an arbitrary region in a second captured image captured by the second imaging unit, the arbitrary region in the first captured image and the arbitrary region in the second captured image are regions in which the same subject exists, the delay instruction means delays the scanning start timing so that a difference in scanning time to the subject approaches zero. Program for.

5. A first imaging unit having a rolling shutter type imaging element; a second imaging unit having a rolling shutter type imaging element having a different resolution from that of the first imaging unit; A computer that controls an imaging apparatus comprising: a delay instruction unit that delays a scanning start timing of one of the first imaging unit and the second imaging unit by a delay time that is a difference in scanning completion timing between the first imaging unit and the second imaging unit in an arbitrary region in a first captured image captured by the first imaging unit and an arbitrary region in a second captured image captured by the second imaging unit, the arbitrary region in the first captured image and the arbitrary region in the second captured image are regions where a vanishing point exists where directions of movement of a stationary object converge on the first captured image and the second captured image, the delay instruction means delays the scanning start timing so that the scanning timings at the vanishing points coincide with each other; Program for.

6. causing the computer to function as a vanishing point detection means for detecting a vanishing point to which directions of movement of a stationary object converge on the first captured image and the second captured image; 6. The program according to claim 5 .

Images