Imaging system and mobile body equipped therewith

Abstract

Provided is an imaging system installed in a moving vehicle, wherein a first image is acquired by an imaging device while blurring is corrected by a blur correction device for a blur correction amount set on the basis of a first distance as a subject distance, and a second image is acquired by the imaging device after the first image while blurring is corrected by the blur correction device for a blur correction amount set on the basis of a second distance as the subject distance. An image processing device calculates the pixel movement amount of a feature portion that is shared by each of the first and the second images. The image processing device detects a third distance as the subject distance on the basis of the pixel movement amount and the movement amount of the moving vehicle.

Description

Technical Field

[0001] The present disclosure relates to an imaging system that corrects blurring in response to the movement of a moving body, and a moving body equipped with the same.

Background Art

[0002] With the aging of transportation infrastructure, the demand for infrastructure inspection is increasing. Instead of visual inspection by humans, by imaging infrastructure facilities while moving with a moving body and detecting defective locations through image processing of the captured images, the inspection efficiency is significantly improved. However, since imaging is performed while moving, blurring occurs in the captured images.

[0003] For example, Patent Document 1 corrects blurring caused by camera movement during exposure using the technology of a saccharide mirror. By irradiating light onto an imaging target, reflecting the light reflected by the imaging target onto a mirror that rotates during a predetermined exposure time, and then making the light incident on a camera, blurring is reduced.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, it is difficult to keep the distance between the imaging device that moves with the moving body and the imaging target constant, and the blurring correction deteriorates due to the change in the distance between the imaging device and the imaging target.

[0006] The present disclosure provides an imaging system that suppresses deterioration of blurring correction even during imaging while moving, and a moving body equipped with the same.

Means for Solving the Problems

[0007] The imaging system of this disclosure is an imaging system installed on a moving object and comprises an imaging device that images at least a portion of the structures surrounding the moving object as the imaging target, and an image correction device that has an image correction amount setting unit that sets an image correction amount based on the subject distance from the imaging device to the imaging target to correct blur in the direction of movement when the imaging device takes an image while the moving object is moving, and corrects blur during imaging using the set image correction amount. A first image is acquired by the imaging device while the image is corrected for blur by the image correction device using an image correction amount set based on a first distance as the subject distance. After the first image, a second image is acquired by the imaging device while the image is corrected for blur by the image correction device using an image correction amount set based on a second distance as the subject distance. The imaging system comprises an image processing device that calculates the pixel movement amount of feature portions common to the first image and the second image, respectively. The image processing device calculates a third distance as the subject distance of a third image to be taken after the second image, based on the pixel movement amount and the movement amount of the moving object.

[0008] Furthermore, the mobile body of this disclosure is equipped with the imaging system described above. [Effects of the Invention]

[0009] The imaging system and mobile body equipped therewith of this disclosure can provide an imaging system and mobile body equipped therewith that suppress the deterioration of image stabilization even when imaging while moving. [Brief explanation of the drawing]

[0010] [Figure 1] Diagram showing a vehicle equipped with an imaging system. [Figure 2] Diagram showing the configuration of the imaging system in the embodiment. [Figure 3A] Diagram explaining image stabilization in the imaging system. [Figure 3B] Diagram illustrating typical subject distances. [Figure 3C] Explanatory diagram showing the subject distance in the embodiment [Figure 4] An explanatory diagram illustrating an image captured without blur correction during exposure. [Figure 5] Explanatory drawing for explaining the movement of feature portions in images respectively captured in consecutive imaging frames [Figure 6] Flowchart showing the blurring imaging process in the embodiment [Figure 7] Graph showing the relationship between the change in moving speed, the timing of exposure time, and the movement blur correction angle [Figure 8] Graph showing the effects of the embodiment [Figure 9] Diagram showing a modification example of the imaging device and the shake correction mechanism [Figure 10] Diagram showing the configuration of the imaging system in the modification example

Mode for Carrying Out the Invention

[0011] (Embodiment) Hereinafter, the embodiment will be described with reference to the drawings. In the embodiment, a case where the moving body is a vehicle 3 such as an automobile and the imaging system 1 is attached to the upper part of the vehicle 3 will be described as an example.

[0012] [1. Configuration of Imaging System] Refer to FIGS. 1 and 2. FIG. 1 is a diagram for explaining the imaging system 1. FIG. 2 is a block diagram showing the internal configuration of the imaging system 1. In FIG. 1, the vehicle 3 is, for example, traveling inside a tunnel 5. On the wall surface 5a inside the tunnel 5, for example, holes 5b and cracks 5c are generated.

[0013] The imaging target of the imaging system 1 is at least a part of the structures around the vehicle 3, and is a target that moves relatively according to the moving speed of the vehicle 3 as the vehicle 3 moves. The imaging target area 9 is an area to be acquired as an image in this imaging target. In addition to the inner wall of the tunnel 5, the side surface and bottom surface of an overpass, utility poles, and electric wires may also be used as imaging targets. Thereby, holes, cracks, bulges, peeling, joints, the inclination of utility poles, and the deflection of electric wires of the imaging target can be detected by image processing from the acquired images.

[0014] Vehicle 3 is equipped with a speed detection device 3a that detects the vehicle's speed. The speed detection device 3a is, for example, a vehicle speed sensor that detects the vehicle's speed from the rotational speed of the vehicle's axle.

[0015] An imaging system 1 is installed on the top surface of vehicle 3. In Figure 1, the imaging system 1 is fixed to capture images of structures above vehicle 3, such as the wall surface 5a of tunnel 5, but it may also be installed to capture images of the wall surface 5a to the side or diagonally to the side of vehicle 3.

[0016] The imaging system 1 comprises an imaging device 11, a blur correction device 13, an image processing device 14, and a storage unit 15. The imaging device 11 captures images of structures around the vehicle 3, and when the vehicle 3 is traveling inside, for example, a tunnel 5, it captures images of the tunnel wall 5a. The imaging device 11 comprises a camera body 21, a lens 23 as an optical lens, a shutter 24, an image sensor 25, and a camera control unit 27.

[0017] The camera body 21 has a replaceable lens 23 attached and houses an image sensor 25 and a camera control unit 27. The image sensor 25 is positioned at the focal length F of the lens 23. The image sensor 25 is a solid-state image sensor such as a CCD image sensor, a CMOS image sensor, or an infrared image sensor. The camera body 21 is positioned on the vehicle 3 such that the orientation of the lens 23 is parallel to the direction of movement of the vehicle 3. For example, the camera body 21 is positioned so that the lens 23 faces forward or backward of the vehicle 3. The camera body 21 and lens 23 may be integrated, for example, as in a video camera, and an image stabilization mechanism 31 is positioned outside the integrated camera body 21 and lens 23. The camera control unit 27 opens the shutter 24 while receiving an exposure instruction signal from the correction processing unit 33. The shutter 24 may be configured with multiple aperture blades opening and closing, or it may be an electronic shutter.

[0018] The image stabilization device 13 corrects the optical path of light incident on the imaging system 1 so that image blur in the target area 9 is reduced even when the imaging device 11 takes images while the vehicle 3 is moving. The image stabilization device 13 comprises an image stabilization mechanism 31 and a correction processing unit 33.

[0019] The image stabilization mechanism 31 corrects the optical path of reflected light L1, which is reflected from ambient light in the imaging target area 9, in accordance with the movement of the vehicle 3. The image stabilization mechanism 31 comprises a mirror 41 and a mirror drive unit 43 that rotates the mirror 41. Alternatively, instead of using a mirror, the image stabilization mechanism 31 may be a pan-tilt mechanism that drives a lens barrel, which integrates the lens 23 and the image sensor 25, around a rotation axis, for example, in the pan and tilt directions. The pan-tilt mechanism has a drive unit that rotates the integrated lens 23 and image sensor 25. The drive unit is, for example, a motor. The image stabilization mechanism 31 may have a tilt function that rotates the camera body 21 and the lens 23 in the vertical direction and a pan function that rotates them in the horizontal direction. Furthermore, the image stabilization mechanism 31 may be a mechanism that rotates the entire imaging device 11, or it may have an optical system lens drive mechanism and an image sensor drive mechanism. In this manner, when image stabilization is performed without using a mirror, the lens 23 and the camera body 21 are positioned so that the orientation of the lens 23 is perpendicular to the direction of movement of the vehicle 3.

[0020] The mirror 41 is rotatably positioned to face the lens 23. The mirror 41 can rotate, for example, in either the forward or counterclockwise direction, and the range of rotational angle may be less than 360 degrees or more than 360 degrees. The mirror 41 totally reflects the light reflected by the object being imaged from the ambient light towards the image sensor 25 of the imaging device 11. The mirror drive unit 43 rotates the mirror 41 from its initial angle to an angle specified to correct image blur, and then returns it to the initial angle after rotating it to the specified angle. The mirror drive unit 43 is, for example, a motor. The rotation angle of the mirror 41 is limited by the mechanical constraints of the mirror drive unit 43, and the mirror 41 can be rotated up to the maximum swing angle of the mirror 41 determined by this limitation.

[0021] The image stabilization provided by the image stabilization device 13 will be explained with reference to Figures 3A and 4. Figure 3A is an explanatory diagram illustrating the image stabilization of the imaging system 1. Figure 4 shows an image captured without image stabilization, with Figure 4(a) showing the image at the start of imaging and Figure 4(b) showing the image at the end of imaging.

[0022] For example, suppose the imaging system 1 located at position A moves to position B with the vehicle 3 after an exposure time Tp. Imaging starts at position A, and the image ImA acquired at this timing is shown in Figure 4(a). Image ImA captures, for example, the hole 5b in the imaging target area 9. However, because the exposure time for image ImA is insufficient, the image is dark and not clear.

[0023] Therefore, exposure continues until vehicle 3 moves to position B. In this case, if no blur correction is performed, the imaging target area 9 moves relative to the direction of vehicle 3's movement, resulting in an image ImAa with the hole 5b moving relative to it, as shown in Figure 4(b). In image ImAa, the amount of pixel movement P is detected as the amount of blur. Thus, the image captured by the imaging device 11 while vehicle 3 is moving is a blurred image.

[0024] Therefore, depending on the movement speed of the imaging system 1 and the vehicle 3, the end of the mirror 41 on the side facing the direction of movement rotates the mirror 41 during the exposure time in a direction that cancels out the relative movement of the object being imaged. This allows the imaging system 1 to capture the same object area 9 in the image during the exposure time, thereby obtaining an image with significantly reduced blur. In Figure 1, the mirror 41 is rotated clockwise so that the end of the mirror 41 on the side facing the direction of movement rotates around the object being imaged during the exposure time. By rotating the mirror 41, the amount of pixel movement P in the image ImAa is corrected to zero.

[0025] Next, the outline of this embodiment will be described. As will be described later, the amount of blur correction during imaging changes according to the magnitude of the pixel movement of feature points included in the imaging target from the start to the end of imaging. The amount of pixel movement is proportional to the subject magnification M in addition to the vehicle movement amount. The subject magnification M is inversely proportional to the subject distance from the image sensor 25 to the imaging target. Therefore, if the subject distance changes, the amount of pixel movement P also changes, and if the amount of blur correction is not set according to the change in subject distance, the accuracy of blur correction during imaging will decrease.

[0026] Here, with reference to Figures 3B and 3C, the subject distance from the image sensor 25 to the object being imaged will be described in detail. Figure 3B is an explanatory diagram illustrating a typical subject distance D. Figure 3C is an explanatory diagram showing the subject distance D in the embodiment.

[0027] As shown in Figure 3B, the subject distance D [m] is the distance from the principal point Lp of the lens Ln, which is positioned between the subject Sb and the image sensor 25, to the subject Sb. The distance from the principal point Lp of the lens Ln to the image sensor 25 is the focal length F of the lens Ln. Depending on the shape of the lens Ln, or if the lens Ln is composed of multiple lenses, the principal point Lp of the lens Ln is not necessarily located at the center of the lens Ln, but may be located outside the lens Ln.

[0028] The imaging range Sa, which represents the size of the subject that fits within the field of view, is determined by the ratio of the similar shapes of two triangular regions Sq1 and Sq2, based on the subject distance D, focal length F, and the vertical size Sc [mm] of the image sensor 25. When parallel light L from the shooting range a enters the lens Ln, it is focused onto the image sensor 25.

[0029] Refer to Figure 3C. In this embodiment, the subject distance D is the distance Dk from the imaging target area 9 to the mirror 41 and the distance Dm from the mirror 41 to the principal point Lp of the lens 23. Therefore, the relationship D = Dk + Dm holds.

[0030] Since the subject distance D fluctuates with the movement of the vehicle 3, the subject distance is detected and updated even while the vehicle is moving, thereby suppressing a decrease in the accuracy of image blur correction during imaging in response to fluctuations in the subject distance. For this detection of the subject distance, for example, the image movement distance at each imaging interval and the movement distance of the vehicle 3 at each imaging interval are used. The imaging interval may be a time interval or a distance interval. In this embodiment, exposure and imaging are performed in synchronization with the pulse update of the vehicle speed signal output from the speed detection device 3a, so imaging is performed at equidistant intervals. Alternatively, instead of synchronizing with the pulse update of the vehicle speed signal, imaging may be performed at a constant frame rate based on the moving speed output from the speed detection device 3a.

[0031] Refer to Figure 2. The correction processing unit 33 controls the image stabilization mechanism 31 to correct for blur. The correction processing unit 33 includes an image stabilization amount setting unit 51, a vehicle movement amount calculation unit 53, a pixel movement amount calculation unit 55, a subject magnification calculation unit 57, a subject distance calculation unit 59, and a pixel resolution calculation unit 61.

[0032] The correction processing unit 33 is a circuit that can be implemented using semiconductor elements or the like. The correction processing unit 33 can be configured using, for example, a microcontroller, CPU, MPU, GPU, DSP, FPGA, or ASIC. The functions of the correction processing unit 33 may be implemented using hardware alone, or they may be implemented by combining hardware and software. The correction processing unit 33 reads data and programs stored in the memory unit 15 and performs various arithmetic operations to realize predetermined functions.

[0033] The image stabilization amount setting unit 51 calculates the mirror swing angle α of the mirror 41 during imaging in the following sequence, based on the vehicle's movement speed V1, the set exposure time Tp, the subject magnification M, and the focal length F of the lens 23.

[0034] The amount of vehicle movement La of vehicle 3 during the exposure time Tp from the start to the end of imaging is calculated from the movement speed V1 and the exposure time Tp using the following equation (1). La[mm] = V1[km / h] × 10 6×Tp[ms] / (60 2 ×10 3 )...Equation (1)

[0035] The amount of movement P of the pixels on the image sensor 25 during the exposure time Tp from the start to the end of imaging is calculated from the amount of vehicle movement La of the vehicle 3 and the subject magnification M using the following equation (2). P[mm] = La[mm] × M···(2)

[0036] Since the amount of pixel movement P causes blurring, the optical path of the light incident on the lens 23 is changed by a motion blur correction angle θ corresponding to the amount of pixel movement P to prevent blurring. The motion blur correction angle θ is calculated from the amount of pixel movement P and the focal length F using the following equation (3). θ[deg]=arctan(P / F)...Equation (3)

[0037] Since the mirror swing angle α is half the magnitude of the motion shake correction angle θ, it is calculated by the following equation (4). α = θ / k ... (Equation 4) Here, k is a conversion coefficient between the mirror swing angle α, which is the mechanism swing angle of the drive mechanism, and the motion blur correction angle θ, which is the optical correction angle at which the light incident on the lens 23 is corrected. In the configuration shown in the embodiment of Figure 1, where light from the object to be imaged travels in the order of mirror 41, lens 23, and image sensor 25, the conversion coefficient k = 2. Also, in the case of a pan-tilt mechanism in which the imaging device 11 and the blur correction mechanism 31 are integrated, or a configuration in which the entire camera is driven, the conversion coefficient k = 1.

[0038] In this way, the image stabilization amount setting unit 51 calculates the mirror swing angle α of the mirror 41.

[0039] Therefore, by rotating the mirror 41 in the opposite direction to the direction of movement during exposure, the imaging device 11 can receive light from the same imaging target area 9 during the exposure time, thereby suppressing motion blur in the captured image. Here, the focal length F in equations (2) and (3) is a value determined by the lens 23, and the subject magnification M is a value determined by the focal length F and the subject distance.

[0040] Furthermore, the parameters used for blur correction are the pixel movement amount, which is the amount of movement of feature points during the imaging interval, and consists of the pixel movement amount qa [px] in pixels and the pixel movement amount qb [mm] in length units, the subject distance D [m], the vehicle movement amount Lb [mm] of the vehicle 3 during the imaging interval, the pixel pitch s [μm / px] of the captured image, and the vertical number of pixels in the captured image Vpx [px]. The focal length F [mm] of the lens 23, the pixel pitch s [μm / px] of the captured image, and the vertical number of pixels in the captured image Vpx [px] are predetermined values. Also, in this embodiment, when both the pixel movement amount qa in pixels and the pixel movement amount qb in length units are referred to, it is simply written as pixel movement amount q.

[0041] Refer to Figure 5. While Figure 4 showed feature points moving during imaging, Figure 5 shows the image ImB of the first feature point captured in an imaging frame at a certain time point, and the image ImC of the first feature point captured in the next imaging frame.

[0042] The image processing device 14 calculates the pixel displacement amount q of a feature point, such as a hole 5b, between imaging frames by performing an image matching process on two captured images ImB and ImC, which were captured in different frames. The pixel displacement amount q may be a single point in the captured images ImB and ImC, or it may be the average value of the respective pixel displacement amounts q1, q2, ... qn at multiple points.

[0043] The image processing device 14 is a circuit that can be implemented using semiconductor elements or the like. The image processing device 14 can be composed of, for example, a microcontroller, CPU, MPU, GPU, DSP, FPGA, or ASIC. The functions of the image processing device 14 may be implemented using hardware alone, or by combining hardware and software. The image processing device 14 reads data and programs stored in the memory unit 15 and performs various arithmetic operations to realize predetermined functions.

[0044] In this embodiment, exposure and imaging are performed in synchronization with the pulse update of the vehicle speed signal output from the speed detection device 3a. This allows the amount of movement of the vehicle 3 to be directly calculated from the vehicle speed signal, instead of detecting the vehicle's moving speed. The vehicle movement amount calculation unit 53 calculates the amount of vehicle movement Lb of the vehicle 3 between the imaging interval of the two captured images, based on the vehicle speed signal received from the speed detection device 3a. The amount of vehicle movement Lb is Lb = a, where a is a constant value and is the distance the vehicle 3 moves during the pulse update.

[0045] The pixel movement amount calculation unit 55 calculates the pixel movement amount qb in units of length using the following equation (5). qb = qa × s / 10 3 ...(5)

[0046] The subject magnification calculation unit 57 calculates the subject magnification M using the following equation (6). M = qb / Lb = qa × s / 10 3 / a···(6)

[0047] The subject distance calculation unit 59 calculates the subject distance D2 using the following equation (7). D = F / M / 10 3 ...(7)

[0048] Here, the vertical imaging range R satisfies the following relation (8). R = Vpx × s / 10 3 ×D / F···(8)

[0049] Therefore, the pixel resolution calculation unit 61 calculates the pixel resolution X using the following equation (9). X = R / Vpx × 10 3 ...(9)

[0050] The pixel resolution calculation unit 61 can associate the pixel units and length units of the captured image from the calculated pixel resolution of the captured image, and may measure the length of cracks in the captured image, or it may store the captured image converted to length units in the storage unit 15.

[0051] The storage unit 15 is a storage medium that stores the programs and data necessary to realize the functions of the correction processing unit 33. The storage unit 15 can be implemented, for example, by a hard disk (HDD), SSD, RAM, DRAM, ferroelectric memory, flash memory, magnetic disk, or a combination thereof.

[0052] The operation unit 7 is an input device for the user to give instructions to the correction processing unit 33. The operation unit 7 may be an input device dedicated to the imaging system 1, or it may be a mobile terminal such as a smartphone. When a mobile terminal is used as the operation unit 7, data is transmitted and received between the operation unit 7 and the correction processing unit 33 via wireless communication. The user may use the operation unit 7 to indicate whether the area to be imaged is a dark area indoors, such as a tunnel, or a bright area outdoors, such as a mountain slope. When imaging is performed at regular time intervals, the user may also specify the frame rate Tf using the operation unit 7. The operation unit 7 also has a display unit that displays warnings from the imaging system 1 to inform the user.

[0053] [2. Operation of the imaging system] Next, the operation of the imaging system 1 will be explained with reference to Figures 6 and 7. Figure 6 is a flowchart showing the imaging process performed by the imaging system 1. Figure 7 is a graph showing the relationship between changes in movement speed, exposure time timing, captured images, and the vehicle speed update flag. Figure 7(a) is a graph showing the movement speed of the vehicle 3 as it progresses over time. Figure 7(b) is a graph showing the exposure time timing for each frame. Figure 7(c) is an explanatory diagram showing the images captured for each frame. Figure 7(d) is a graph showing the timing when the vehicle speed update flag is raised. The imaging process shown in Figure 6 is started, for example, when the operation unit 7 instructs the system to start imaging while the vehicle 3 is moving.

[0054] When the control unit 7 instructs the system to start imaging, the imaging process for the first frame, image Im0, begins. The blur correction amount setting unit 51 of the correction processing unit 33 calculates and sets the motion blur correction angle θ, which is the blur correction amount, based on a predetermined distance da1 stored in the storage unit 15 as the subject distance D1 (step S1). Immediately after the start of imaging, the subject distance D2 has not been calculated from the captured image, so either the predetermined distance da1 is used, or, for example, a value input from the control unit 7 may be used. Hereinafter, the set or updated subject distance value will be referred to as D1, and the calculated subject distance value as D2.

[0055] The correction processing unit 33 calculates the mirror swing angle α based on the set motion blur correction angle θ, and calculates the mirror rotation speed based on the calculated mirror swing angle α and the set exposure time Tp. The correction processing unit 33 further rotates the mirror 41 at the calculated mirror rotation speed so that the mirror 41 starts rotating from a predetermined initial angle. This performs blur correction during imaging by the imaging device 11 (step S2). At the same time, the correction processing unit 33 continues to send a Hi signal to the camera control unit 27 to instruct exposure for the duration of the exposure time Tp.

[0056] In the imaging device 11, the camera control unit 27 opens the shutter 24 to expose the image Im0 of the first frame while receiving a Hi signal, and the correction processing unit 33 obtains information on the movement of the vehicle 3 between imaging frames from the speed detection device 3a (step S3).

[0057] Once the exposure time Tp has elapsed, the correction processing unit 33 continues to send a Low signal to the camera control unit 27 to instruct it to stop exposure. While the camera control unit 27 is receiving the Low signal, it closes the shutter 24, and the correction processing unit 33 instructs the mirror drive unit 43 to rotate the mirror 41 in the reverse direction to return the mirror 41 to its initial angle. Alternatively, the mirror drive unit 43 may rotate the mirror 41 in the forward direction to return the mirror 41 to its initial angle.

[0058] The correction processing unit 33 determines whether the number of acquired images is two or more (step S4). If it determines that the number of acquired images is not two or more (No. in step S4), it returns to step S1 and starts the imaging process for the next frame, for example, the second frame, image Im1.

[0059] In the imaging process for the second frame, the blur correction amount setting unit 51 sets the motion blur correction angle θ, which is the blur correction amount, based on the same distance da1 as the first frame, as the subject distance D1. After this, the same imaging process as the first frame is performed, and the imaging device 11 captures the first image Im1 of the second frame, as shown in Figure 7. The captured image Im1 is stored in the storage unit 15.

[0060] Since two images Im0 and Im1 are stored in the storage unit 15, when the correction processing unit 33 determines that there are two or more captured images (Yes in step S4), the pixel movement amount calculation unit 55 calculates the pixel movement amount qb between the two or more captured images, and the vehicle movement amount calculation unit 53 calculates the vehicle movement amount Lb of the vehicle 3 that moved from the start of capturing the current image frame to the start of capturing the next image frame based on the acquired movement information (step S5). The calculated vehicle movement amount Lb is stored in the storage unit 15 in association with the captured images. The subject distance calculation unit 59 calculates the subject distance D2 based on the pixel movement amount qb between the two or more captured images and the vehicle movement amount Lb (step S6).

[0061] Image Im1 and Image Im0 each capture a common feature portion g1, and the image processing device 14 detects the pixel displacement amount q1 between Image Im0 and Image Im1 through image matching processing. Based on this pixel displacement amount q1, the subject distance calculation unit 59 calculates the distance ds2 as a new subject distance D2.

[0062] When the subject distance calculation unit 59 calculates a new subject distance D2, it updates the subject distance D1 to the subject distance D2 (step S7). For example, the subject distance D1 calculated based on the nth frame image is updated as the subject distance D1 for the (n+1)th frame. Here, the updated subject distance D1 (=ds2) is used to set the amount of blur correction when capturing the image for the next frame.

[0063] If the correction processing unit 33 determines that imaging is complete (Yes in step S8), it terminates imaging by the imaging system 1 while the vehicle 3 is moving. If the correction processing unit 33 determines that imaging is not complete (No in step S8), it returns to step S1 to capture an image in the next frame.

[0064] Returning to step S1, the next frame, for example, the third frame image Im2, is captured. During the imaging process for the third frame, the blur correction amount setting unit 51 sets the motion blur correction angle θ, which is the blur correction amount, based on a second distance ds2, which is different from the subject distance D1 used for the second frame. After this, the same imaging process as for the second frame is performed, and the imaging device 11 captures the second image Im2 of the third frame, as shown in Figure 7. The captured image Im2 is stored in the storage unit 15.

[0065] When imaging is performed continuously, the image processing device 14 detects the pixel movement amount q2 of a common feature portion g1 captured in the first image Im1, which is captured based on a first distance (e.g., ds1), and the second image Im2, which is captured based on a second distance (e.g., ds2). Based on this pixel movement amount q2, the subject distance calculation unit 59 calculates the distance ds3 as a new subject distance D2. Furthermore, by updating this as the subject distance D1 of the next captured image Im3 and capturing the image, the motion blur correction angle θ can be set accurately according to the changing subject distance, thereby improving the accuracy of blur correction. In addition, the subject distance D1 changes when the ratio of the pixel movement amount q to the vehicle movement amount Lb of the vehicle 3 changes.

[0066] In this way, two effects of the imaging system 1 of the embodiment that updates the subject distance will be explained. The two effects are the effect of improving the accuracy of calculating the subject distance D and the effect of responding to fluctuations in the subject distance D. The effect of improving the accuracy of calculating the subject distance D is that, for example, simply processing two consecutive images once and updating the subject distance D once may not provide sufficient accuracy. Therefore, it may be necessary to update the subject distance D even when the fluctuation is small.

[0067] If the subject distance D cannot be set accurately, the blur correction accuracy of the captured image decreases, which in turn decreases the accuracy of the movement amount calculation by image processing, and consequently the accuracy of the subject distance D calculation also decreases. Thus, improving the blur correction accuracy and improving the accuracy of the subject distance D calculation are interrelated, so it is not possible to improve one accuracy first and then the other. However, with regard to image processing, it is possible to detect the amount of pixel movement even if the blur is not zero. For example, as shown in Figure 8, images Im1 and Im2 are captured with blur correction using a first distance ds1 which is a hypothetically set subject distance, and a second distance ds2 is calculated as the subject distance by image processing based on these images. An image Im3 is captured with blur correction using the calculated second distance ds2, and a third distance ds3 is calculated as the subject distance by image processing based on images Im2 and Im3. By using the calculated third distance ds3, an image Im4 with even greater accuracy of blur correction can be captured. Therefore, each time the subject distance D is updated, the calculated subject distance approaches the true value Dt, which allows for the capture of images with improved blur correction accuracy, and further improves the accuracy of movement calculation through image processing.

[0068] Furthermore, as already mentioned above, the effect of responding to changes in the subject distance D is that even if the true value Dt of the subject distance changes, the calculated subject distance will approach it with each update, so blur correction can be performed in response to changes in the subject distance.

[0069] [3. Effects, etc.] Thus, the imaging system 1 includes an imaging device 11 having an image sensor 25 that images an object to be imaged, which is located away from the vehicle 3 and is at least a part of the area surrounding the vehicle 3, and an image blur correction device 13 having a vehicle movement amount calculation unit 53 that calculates the amount of movement of the vehicle 3, and which sets a blur correction amount based on the subject distance D1 from the imaging device 11 to the object to be imaged, and corrects the blur during imaging using the set blur correction amount. The system further includes an image processing device 14 that calculates the pixel movement amount q of a common feature portion g1 in a first image Im1 captured by the imaging device 11 and a second image Im2 captured after the first image Im1. The subject distance calculation unit 59 of the image stabilization device 13 calculates a third distance ds3 as the subject distance D2 of the third image Im3, which is captured after the second image Im2, based on the pixel movement amount q and the vehicle movement amount Lb of the vehicle 3 during the imaging interval between the first image Im1, where the first distance ds1 is the subject distance D1, and the second image Im2, where the second distance ds2 is the subject distance D1.

[0070] Even when the actual subject distance Dt changes as a true value due to the movement of vehicle 3, the accuracy of image stabilization can be improved by detecting the subject distance D1 and calculating the subject distance D2 while vehicle 3 is moving.

[0071] Furthermore, the image stabilization device 13 updates the subject distance D1 to a third distance ds3 and sets the amount of image stabilization when capturing the third image Im3 based on the updated subject distance D1. Since the amount of image stabilization is updated based on the updated subject distance D1, it is possible to capture a third image Im3 with improved image stabilization.

[0072] Furthermore, the first distance ds1 and the second distance ds2, which are the subject distances when capturing the first image Im1 and the second image Im2, may be values ​​detected based on the amount of pixel movement in the two previously captured images and the amount of movement of the moving object during the interval between the capture of the two images.

[0073] Furthermore, immediately after the start of imaging, the subject distance D1 of the initial image Im0 and the subject distance D1 of the second image Im2 may be the same predetermined value ds1.

[0074] Furthermore, the first distance ds1 and the second distance ds2 are values ​​detected based on the pixel displacement amounts q1 and q2 of images Im1 and Im2, which were captured before the third image Im3, and the vehicle displacement amount Lb of vehicle 3, respectively.

[0075] Furthermore, the image stabilization device 13 may update the subject distance D1 by calculating the average value of the subject distance calculated two frames prior and the subject distance calculated one frame prior. For example, when capturing the fourth image Im4, the subject distance D2 may be calculated by calculating the average value of the distance ds2 calculated two frames prior and the distance ds1 calculated one frame prior, and the subject distance D2 calculated as the average value may be updated as the subject distance D1.

[0076] (Other embodiments) As described above, the above embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted as appropriate. Therefore, other embodiments will be illustrated below.

[0077] In the above embodiment, the amount of pixel movement of feature points obtained by image matching of two consecutive captured images was used to calculate the subject distance of the third image, but this is not limited to this. If the image matching process of the two captured images is delayed compared to the imaging cycle, the subject distance may be calculated with a delay of several frames.

[0078] In the above embodiment, the image processing device 14 may output a warning signal to, for example, the operation unit 7, the speaker, or the image display device if the detected second distance exceeds a predetermined range. This allows the driver of the vehicle 3 to recognize that the second distance exceeds a predetermined range. A warning signal is output when the subject distance is shorter or longer than expected.

[0079] If the subject distance is shorter than expected, the subject magnification M increases, and the amount of pixel movement also increases. This increases the swing angle of the mirror required for image stabilization, which may exceed the maximum swing angle. Also, if the subject distance is longer than expected, the subject magnification M decreases, and the pixel resolution [mm / px], which is the actual size of each pixel, becomes coarser, which may prevent the required inspection accuracy from being achieved. In this way, if the second distance exceeds a predetermined range, the deterioration of image stabilization may not be reduced, or the required inspection accuracy may not be ensured. By warning the user, the user can re-inspect. For example, if the subject distance changed due to vehicle 3 meandering, a re-inspection can be performed by driving the vehicle again. Also, if there is no problem with the vehicle 3's movement but the problem is due to a change in the subject, the user can be prompted to change lens 23.

[0080] In the above embodiment, the imaging device 11 and the image stabilization mechanism 31 were separate components, but this is not limited to this configuration. As shown in Figure 9, the imaging device 11 may have a housing 22 that connects the lens 23 and the camera body 21, and the image stabilization mechanism 31 may be housed inside the housing 22.

[0081] In the above embodiment, a motion blur correction angle θ corresponding to the amount of pixel movement P was calculated for blur correction, and the mirror 41 was rotated by a mirror swing angle α corresponding to the motion blur correction angle θ to perform imaging, but the embodiment is not limited to this. Instead of the motion blur correction angle θ, the image sensor 25 and lens 23 may be driven in translation for blur correction. In this case, as shown in the imaging system 1B in Figure 10, the lens 23 of the imaging device 11 is directed directly towards the imaging target area 9 (see Figure 1), and the blur correction mechanism 31B has an image sensor drive mechanism 71 that drives the image sensor 25 in translation with respect to the imaging target area 9, and an optical system lens drive mechanism 73 that drives the lens 23 in translation. The image sensor drive mechanism 71 and the optical system lens drive mechanism 73 are, for example, translation drive mechanisms composed of a pinion gear, a rack, and a motor. The translational drive amount used by the image sensor drive mechanism 71 to drive the image sensor 25 is equal to the pixel movement P, and the translational drive amount used by the optical system lens drive mechanism 73 to drive the lens 23 is the pixel movement P multiplied by a constant m. The constant m is a number greater than or equal to 0. The translational amount relative to the lens drive amount changes depending on the optical system. Alternatively, instead of individually translating the lens 23 and the image sensor 25, the entire imaging device 11 may be driven translationally.

[0082] In the above embodiment, information on the moving speed V1 from the vehicle 3's speed detection device 3a was used, but the embodiment is not limited to this. Furthermore, the imaging system 1 may be equipped with a speed detection device that detects the moving speed of the imaging system 1. The speed detection device may also utilize a GPS (Global Positioning System) system or an acceleration sensor.

[0083] In the above embodiment, the imaging system 1 was imaging the upper and side walls of the vehicle 3, but it is not limited to this. The imaging system 1 may also be imaging the road surface below the vehicle 3. From the captured images, potholes, cracks, rutting, etc., that have occurred on the road surface can be detected by image processing.

[0084] In the above embodiment, the case where the moving object is a vehicle 3 such as an automobile was described. However, the moving object is not limited to a vehicle 3, but may also be a vehicle that travels on land such as a train or motorcycle, a ship that travels on the sea, or an aircraft that flies in the air such as an airplane or drone. When the moving object is a ship, the imaging system 1 images the bottom surface of bridge piers and bridge girders, or structures built along the shore. When the moving object is a train, the deflection of the overhead wires can be detected by imaging the overhead wires. Furthermore, since the vehicle movement amount calculation unit 53 calculates the amount of movement of the vehicle 3 as the moving object, when the moving object is something other than a vehicle, the vehicle movement amount calculation unit 53 calculates the amount of movement of the moving object.

[0085] In the above embodiment, an image was captured using light reflected from ambient light onto the imaging target area 9, but this is not limited to this. Light may be shone from a moving object toward the imaging target area 9, and an image may be captured using the reflected light of the shone object.

[0086] (Summary of the embodiment) (1) The imaging system of the present disclosure is an imaging system installed on a moving body and comprises an imaging device having an image sensor for imaging an object that is located away from the moving body and is at least a part of the area surrounding the moving body, and an image blur correction device having a moving body movement amount calculation unit for calculating the amount of movement of the moving body, and which sets an image blur correction amount based on the subject distance from the imaging device to the object to be imaged for blur in the direction of movement when the imaging device is imaging while the moving body is moving, and corrects blur during imaging using the set image blur correction amount. The imaging system comprises an image processing device that calculates the pixel movement amount of common feature portions in a first image captured by the imaging device and a second image captured after the first image. The image blur correction device calculates a third distance as the subject distance of a third image to be captured after the second image, based on the pixel movement amount and the amount of movement of the moving body at the imaging interval between the first image with a first distance as the subject distance and the second image with a second distance as the subject distance.

[0087] This allows the camera to detect the subject distance even while the moving object is in motion, enabling image stabilization to be performed in response to changes in the subject distance, thereby improving the accuracy of image stabilization.

[0088] (2) In the imaging system of (1), the image stabilization device updates the subject distance to a third distance and sets the amount of image stabilization when capturing the third image based on the updated subject distance.

[0089] (3) In the imaging system of (1) or (2), the first distance and the second distance, which are the subject distances when capturing the first image and the second image, are values ​​detected based on the amount of pixel movement of the two previously captured images and the amount of movement of the moving object during the interval between the capture of the two images.

[0090] (4) In the imaging system of (1) or (2), immediately after the start of imaging, the first distance and the second distance are the same predetermined value.

[0091] (5)(2) In the imaging system, the image stabilization device updates the subject distance when capturing the third image by the average value of the second distance calculated when capturing the second image and the third distance calculated when capturing the third image.

[0092] (6) In any of the imaging systems described in (1) to (5), the subject distance changes when the ratio of pixel movement to the movement of the moving object changes.

[0093] (7) In any of the imaging systems described in (1) to (6), the image processing device calculates the pixel resolution of the second image based on the second distance, and converts the predetermined pixel size units in the second image from pixel units to length units based on the calculated pixel resolution.

[0094] (8) In the imaging system of (2) or (5), the image processing device calculates a second pixel movement amount for feature portions common to the second image and the third image, respectively. The image stabilization device calculates a fourth distance as the subject distance based on the second pixel movement amount and the amount of movement of the moving object during the imaging interval between the second and third images. The image stabilization device updates the subject distance to the fourth distance and sets the amount of image stabilization when capturing the fourth image, which is captured after the third image, based on the updated subject distance. By repeatedly updating the subject distance in this way, the accuracy of the calculated subject distance can be improved, and the accuracy of image stabilization can be further improved.

[0095] (9) In any of the imaging systems described in (1) through (8), the image processing device outputs a warning signal if the detected second distance exceeds a predetermined range.

[0096] (10) In any of the imaging systems described in (1) to (9), a speed detection device is provided to detect the movement speed of the imaging system.

[0097] (11) In any of the imaging systems described in (1) to (10), the image stabilization device comprises a mirror that totally reflects the ambient light reflected by the object being imaged toward the imaging device, and a mirror drive unit that rotates the mirror, wherein the amount of image stabilization corresponds to the rotation angle of the mirror.

[0098] (12) In any of the imaging systems described in (1) to (10), the imaging device has an optical lens integrated with an image sensor, and the image stabilization device has a drive unit that rotates the integrated optical lens and the image sensor, and the amount of image stabilization corresponds to the rotation angle of the drive unit.

[0099] (13) In any of the imaging systems described in (1) to (10), the imaging device has an optical lens focused on an image sensor, and the image stabilization device includes an optical lens and a translational drive mechanism that translates the image sensor in the direction within the imaging plane of the image sensor, and the amount of image stabilization corresponds to the translation amount of the optical lens.

[0100] (14) The mobile body of the present disclosure comprises any of the imaging systems (1) to (13). This allows the imaging system to capture images of the area around the mobile body with reduced blur while the mobile body is moving.

[0101] The imaging system described in this disclosure is realized through hardware resources, such as a processor, memory, and cooperation with programs. [Industrial applicability]

[0102] This disclosure is applicable to imaging systems installed on moving objects. [Explanation of Symbols]

[0103] 1. Imaging System 3 vehicles 3a Speed ​​detection device 5 tunnels 5a Wall surface 5b hole 5c crack 7 Control section 9. Area to be imaged 11 Imaging device 13 Image stabilization device 14 Image Processing Device 15 Storage section 17 Correction Processing Unit 21 Camera body 23 lenses 24 shutters 25 Image sensor 27 Camera Control Unit 31 Image stabilization mechanism 33 Correction Processing Unit 41 Mirror 43 Mirror drive unit 51 Image stabilization amount setting section 53 Vehicle movement calculation unit 55 Pixel movement amount calculation unit 57 Subject magnification calculation section 59 Subject distance calculation section 61 Pixel resolution calculation unit 71 Image sensor drive mechanism 73 Optical lens drive mechanism α Mirror Swing Angle F focal length M Subject magnification V1 Movement speed

Claims

1. An imaging system installed on a mobile device, An imaging device having an image sensor that captures an image target which is at least a part of the area surrounding the moving body, away from the moving body; A motion correction device has a motion amount calculation unit that calculates the amount of motion of the motion object, and sets a motion correction amount based on the subject distance from the imaging device to the object to be imaged when the imaging device takes an image while the motion object is moving, and corrects the motion during image capture using the set motion correction amount. The system comprises an image processing device that calculates the pixel displacement amount of a common feature portion in a first image captured by the imaging device and a second image captured after the first image, respectively. The image stabilization device calculates a third distance as the subject distance for a third image to be captured after the second image, based on the amount of pixel movement and the amount of movement of the moving body during the imaging interval between the first image, where the first distance is the subject distance, and the second image, where the second distance is the subject distance. Imaging system.

2. The image stabilization device updates the subject distance to the third distance and sets the amount of image stabilization when capturing the third image based on the updated subject distance. The imaging system according to claim 1.

3. The first distance and the second distance, which are the subject distances when capturing the first and second images, are values ​​detected based on the pixel movement amount of the two previously captured images and the movement amount of the moving object during the interval between the capture of the two images. The imaging system according to claim 1 or 2.

4. Immediately after the start of imaging, the first distance and the second distance are the same predetermined value. The imaging system according to claim 1 or 2.

5. The image stabilization device updates the subject distance when capturing the third image by using the average value of the first distance calculated when capturing the first image and the second distance calculated when capturing the second image. The imaging system according to claim 2.

6. When the ratio of the pixel movement amount to the movement amount of the moving body changes, the subject distance changes. The imaging system according to claim 1.

7. The image processing device calculates the pixel resolution of the second image based on the second distance, Based on the calculated pixel resolution, the predetermined unit of pixel size in the second image is converted from pixel units to length units. The imaging system according to claim 1.

8. The image processing device calculates a second pixel movement amount for a feature portion common to the second image and the third image, The image stabilization device calculates a fourth distance as the subject distance based on the second pixel movement amount and the movement amount of the moving object during the imaging interval between the second and third images. The image stabilization device updates the subject distance to the fourth distance and sets the amount of image stabilization when capturing the fourth image, which is captured after the third image, based on the updated subject distance. The imaging system according to claim 2 or 5.

9. The image processing device outputs a warning signal when the detected second distance exceeds a predetermined range. The imaging system according to claim 1.

10. The system includes a speed detection device for detecting the movement speed of the imaging system. The imaging system according to claim 1.

11. The aforementioned image stabilization device, A mirror that totally reflects the ambient light reflected by the object being imaged back towards the imaging device, The system includes a mirror drive unit that rotates the aforementioned mirror, The amount of shake correction corresponds to the rotation angle of the mirror, The imaging system according to claim 1.

12. The imaging device has an optical lens integrated with the image sensor, The image stabilization device includes a drive unit that rotates the integrated optical lens and the image sensor, The amount of shake correction corresponds to the rotation angle of the drive unit, The imaging system according to claim 1.

13. The imaging device has an optical lens focused on the image sensor, The image stabilization device includes a translation drive mechanism that translates the optical lens and the image sensor in the direction within the imaging plane of the image sensor. The aforementioned image stabilization amount corresponds to the translation amount of the optical lens. The imaging system according to claim 1.

14. A mobile body comprising the imaging system according to claim 1.

Images