Image processing device

Abstract

Image processing device (21) comprising: an on-board camera (5) that takes a picture of a driving environment in front of a vehicle (1); an image recognition processor (22) that generates a distance image within an image recognition area based on the image captured by the onboard camera (5); an object recognition processor (25) that recognizes an object within the image recognition area and a distance to the object based on the image captured by the onboard camera (5) and the distance image generated by the image recognition processor (22); and an image mode switching unit (24) that switches the image recognition area to be recognized by the object recognition processor (25), wherein The image mode switching unit (24) checks whether a collision with a pedestrian is detected and an activation command signal is issued to a pedestrian protection mechanism, and In a case where an output of the activation command signal is detected, the image mode switching unit (24) switches from a normal mode in which a normal effective image area (Ie) is set as the image recognition area to an area limitation mode in which an image area in which the pedestrian protection mechanism does not appear is set as the image recognition area.

Description

CROSS-REFERENCE TO RELATED REGISTRATIONS

[0001] The present application claims priority from Japanese patent application No. 2016-009059, filed on January 20, 2016, the entire contents of which are hereby incorporated by reference. BACKGROUND 1. Technical field

[0002] The present invention relates to an image processing device which allows the resumption of a driver assistance control based on a camera image without stopping, even when a pedestrian protection mechanism is activated. 2. Related technology

[0003] If a pedestrian collides with the front bumper of a vehicle traveling at a predetermined speed or higher, the front shock absorber will knock the pedestrian's legs upwards, causing the pedestrian's upper body to fall onto the hood, and the pedestrian's head will secondarily collide with, for example, the windscreen or the front pillars (A-pillars).

[0004] Pedestrian protection airbags are known as a countermeasure to avoid such secondary collisions. With regard to such a pedestrian protection airbag, if a collision with a pedestrian is detected, an airbag located in the engine compartment is inflated and deployed towards the windshield and the front pillars (A-pillars) to cover these rigid sections and thereby prevent the pedestrian's head from directly colliding with them. In another known technique for avoiding a secondary collision, if a pedestrian collides with a moving vehicle, the hood is raised so that it acts as a damping element.

[0005] In a vehicle equipped with an onboard camera that detects the front of the vehicle, the activation of a pedestrian protection mechanism, such as a pedestrian airbag or pop-up hood, may unintentionally cause part of the airbag or hood to enter the camera's field of view. If this part of the airbag or hood is mistakenly identified as an obstacle located at a very short distance, there is a possibility of erroneous activation of the driver assistance systems, such as autonomous emergency braking (AEB). Therefore, when the pedestrian protection mechanism is activated, the driver assistance systems are automatically deactivated based on the image captured by the onboard camera.

[0006] However, if the driver assistance control system stops, the driver might attempt to avoid a collision with the pedestrian and make an erroneous driving maneuver (e.g., erroneous steering and acceleration), potentially making it impossible to prevent a secondary collision (such as a collision with a utility pole, driving over a road shoulder, etc.). For example, the unexamined Japanese patent application JP 2004-284440A discloses a technique in which, even if the airbag or the front hood blocks the windshield and obstructs the driver's view, this is adequately addressed by ensuring the driver's view is maintained with an image captured by a camera positioned in front of the rearview mirror.

[0007] Furthermore, DE 602 07 655 T2, for example, discloses a device for displaying the surroundings of a vehicle, comprising: a camera that captures an image of the surroundings of a vehicle; an obstacle detection device that detects an obstacle in the surroundings of the vehicle; an image processing device that generates an image of the surroundings, representing the situation around the vehicle, from the image captured by the camera, characterized by: a device for detecting missing image areas that detects a missing area in the image of the surroundings when the obstacle is detected by the obstacle detection device, wherein the missing image area cannot obtain image data from the camera due to the presence of the obstacle;wherein the device for detecting missing image areas includes a device for detecting object positions which detects the position of a boundary of the obstacle on the side of the vehicle and determines the missing image area using the detected boundary position, wherein the image processing device uses a boundary image which contains at least the boundary section of the obstacle on the side of the vehicle to generate substitute image data, and fills at least a part of the missing image area with the substitute image data when the missing image area is detected.

[0008] DE 10 2007 014 012 A1 discloses a vehicle environment monitoring device, a vehicle environment monitoring method, and a vehicle environment monitoring program that can rapidly identify an object to be avoided, such as a pedestrian, from an image of the vehicle's surroundings and provide information to the driver or control the vehicle's behavior. The vehicle environment monitoring device includes an object extraction processing unit that extracts objects present around a vehicle from images captured by infrared cameras, a pedestrian extraction processing unit that extracts a pedestrian from the extracted objects, a posture determination processing unit that determines the posture of the extracted pedestrian, and an avoidance object determination processing unit that uses a determination algorithm to...which includes at least one initial determination processing step regarding the pedestrian's posture determined by the posture determination processing unit, determines whether the extracted object is an object to be avoided, which must be prevented from coming into contact with the vehicle, and a vehicle equipment control processing unit which controls the vehicle's equipment at least according to the determination result of the avoidance object determination processing unit.

[0009] DE 10 2009 012 917 A1 discloses an obstacle detection device that can be installed in a vehicle. This device comprises a camera and processes the resulting image data to detect objects as potential obstacles. A size value, for example, the distance between two rear lights, transmitted by a vehicle ahead, and a corresponding size value measured in an image captured by the camera, are used to calculate the distance to the vehicle ahead. This value, together with the height value transmitted by the vehicle ahead and a corresponding height value measured in the captured image, is used to calculate a relative height difference between the vehicles. This difference is then used to distinguish between detected objects that represent actual obstacles and objects that are not obstacles, such as lane markings.

[0010] DE 60 2005 001 627 T2 Pedestrian detection device for detecting a pedestrian from an input image captured by an image capture device, based on a pattern match of the input image and model images of the pedestrian, the device comprising: an image recognition device for potential pedestrians for detecting a whole-body area surrounding a whole-body model image and a potential partial area surrounding a partial model image of the whole body from the input image based on a pattern match using the whole-body model image and the partial model image of the pedestrian; and a pedestrian image recognition device; wherein: the pedestrian image recognition device, if a potential partial area is present in the whole-body area detected by the image recognition device for potential pedestrians, determines that a detected potential partial area is part of the pedestrian;The pedestrian image recognition device, if no whole-body area has been detected by the potential pedestrian image recognition device, compares a previous potential sub-area extracted from a previous input image with a potential sub-area extracted from the current input image and determines that the potential sub-area extracted from the current input image is part of the pedestrian if the positional distance between the two potential sub-areas is equal to or less than a predetermined distance.

[0011] However, in the technology disclosed in JP 2004 - 284 440 A, the positional relationship between the camera and the airbag and the hood must be adjusted so that the airbag and the hood do not enter the camera's field of view, even if the airbag or the hood blocks the front of the windshield.

[0012] Consequently, implementing the adjustment by limiting the camera's field of view would cause a problem for normal driver assistance control, and furthermore, implementing the adjustment by limiting the inflation and deployment of the airbag or the raising of the hood would be problematic in that it would add restrictions to avoid a secondary collision with a pedestrian in the event of a collision. SUMMARY OF THE INVENTION

[0013] It is desirable to specify an image processing device that not only allows the resumption of driver assistance control based on an image captured by an on-board camera, without limiting the operation of a pedestrian protection mechanism in order to efficiently protect a pedestrian in the event of a collision, but can also prevent a secondary collision of a vehicle.

[0014] One aspect of the present invention provides an image processing device comprising an onboard camera, an image recognition processor, an object recognition processor, and an image mode switching unit. The onboard camera captures an image of the driving environment in front of a vehicle. The image recognition processor generates a distance image within an image recognition area based on the image captured by the onboard camera. The object recognition processor detects an object within the image recognition area and its distance based on the image captured by the onboard camera and the distance image generated by the image recognition processor. The image mode switching unit toggles the image recognition area so that it is recognized by the object recognition processor.The image mode switching unit checks whether a collision with a pedestrian is detected and an activation command signal is sent to a pedestrian protection mechanism. If an activation command signal is detected, the image mode switching unit switches from a normal mode, in which the entire effective image area is set as the image detection area, to an area limitation mode, in which the image area in which the pedestrian protection mechanism does not appear is set as the image detection area. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a perspective view of the front of a vehicle body in a state where a pedestrian protection airbag is being inflated and deployed; Fig. Figure 2 shows the configuration of relevant parts of a pedestrian protection device and an image processing device; Fig. Figure 3 shows an image mode switching process routine in the flowchart; Fig. Figure 4 in the flowchart shows a range limit mode routine; Fig. Figure 5 in the flowchart shows an area limitation mode according to a different procedure; Fig. Figure 6 shows an image taken by an onboard camera; and Fig. 7A shows a current image of an effective image area, Fig. Figure 7B shows a distance image of the effective image area, and Fig. 7C shows the number of effective spacing pixels from each row of the in Fig. Distance image shown in 7B. DETAILED DESCRIPTION

[0015] An example of the present invention will now be described with reference to the drawings. A vehicle body front 2 of a vehicle in Fig. The vehicle 1 shown in Figure 1 is equipped with an engine compartment (not shown). A front bumper 3 is arranged in front of this engine compartment, and the top of the engine compartment is covered by an opening and closing front hood 4.

[0016] An onboard camera 5 is located on the inside of the upper portion of the front windshield 6 of the vehicle. This onboard camera 5 captures an image of the driving environment in front of the vehicle 1 and is a color camera, comprising a main camera 5a and an auxiliary camera 5b. Cameras 5a and 5b are positioned at left and right positions, respectively, equidistant from the center in the width direction of the vehicle. The main camera 5a captures a reference image (right image) required for stereo processing, and the auxiliary camera 5b captures a comparison image (left image) for this process.

[0017] An airbag module 7, serving as a pedestrian protection mechanism, is arranged on the inner surface of the front hood 4. The airbag module 7 contains a pedestrian protection airbag (hereinafter simply referred to as "airbag") 7a and an inflator 7b (see Figure 7). Fig. 2) for deploying the airbag 7a. The front hood 4 opens forward and opens and closes around hood hinges (not shown) located on opposite broadsides at the rear of the front hood 4. When the airbag 7a is to be inflated and deployed, a hood actuator 9 provided on the hood causes the rear of the front hood 4 to spring upward, causing the airbag 7 to inflate and deploy through the resulting gap, as shown in Fig. 1 shown.

[0018] The inflated and deployed airbag 7a covers rigid sections, such as the lower portion of the windshield 6, the left and right A-pillars 8, the front fascia, and the windshield wipers (not shown). Furthermore, when inflated and deployed, the airbag 7a has a U-shaped clearance 7c in its center, and the bottom of this clearance 7c serves as the upper edge of the airbag 7a. This clearance 7c ensures the driver's field of vision and that of the onboard camera 5.

[0019] As in Fig. As shown in Figure 2, the inflator 7b provided on the airbag module 7 and the hood actuator 9 are activated on the basis of a command signal from the pedestrian protection airbag unit 11, which represents a pedestrian protection device.

[0020] In a state where cameras 5a and 5b are synchronized, the onboard camera 5 outputs analog R, G, and B images, which are then converted into digital images with predetermined grayscale levels by an A / D converter 10 and output to an image recognition unit 21. The pedestrian protection airbag unit 11 and the image recognition unit 21, which constitutes the image processing device, are primarily composed of microcomputers and each has, for example, a central processing unit (CPU), a read-only memory (ROM), and a random-access memory (RAM). The ROM stores, for example, various program types to be executed by the CPU, as well as static data.

[0021] The pedestrian protection airbag unit 11 includes a pedestrian collision detection unit 12 and an airbag activation command unit 13 as airbag activation functions to be processed by the CPU. The pedestrian collision detection unit 12 determines a collision with a pedestrian based on a vehicle speed detected by a vehicle speed sensor 14 and a pressure change detected by a collision detection sensor 15. The collision sensor 15 is, for example, a pressure sensor connected to a long pressure tube or pressure chamber located along and within the front bumper 3, extending left and right in the width direction of the vehicle. The collision sensor 15 detects a change in collision pressure when the pressure tube or pressure chamber is compressed as a result of a collision.

[0022] If the vehicle speed is higher than or equal to a pedestrian collision detection vehicle speed, and the collision pressure detected by the collision sensor 15 is within a preset pressure range for detecting a collision with a pedestrian (a minor collision, such as when the front bumper 3 is slightly dented), the pedestrian collision detection unit 12 determines that a collision with a pedestrian has occurred. When the pedestrian collision detection unit 12 determines that a collision with a pedestrian has occurred, the airbag activation command unit 13 outputs a drive signal to the hood actuator 9 and also outputs an activation command signal for inflator 7b to the airbag module 7.

[0023] Then the hood actuator 9 causes the rear of the hood 4 to spring upwards and open, and the airbag module 7 activates the inflator 7b, introducing gas into the airbag 7a. As a result, the airbag 7a inflates and deploys, expanding left and right towards the windshield 6 at the rear of the hood 4, which is opened by the hood actuator 9, so that the airbag module 7 covers the rigid sections, such as the left and right front pillars (A-pillars) 8, which are located in Fig. 1 shown, and the front fairing and windscreen wipers (not shown), to prevent the pedestrian's head from colliding directly with these rigid sections.

[0024] The image recognition unit 21 contains an image recognition processor 22, a camera control unit 23, an image mode switching unit 24, and an object recognition processor 25 as image recognition functions to be processed by the CPU. The image recognition processor 22 detects objects in front of the vehicle 1 based on image data captured by the onboard camera 5. The objects include three-dimensional objects such as a vehicle ahead, a traffic sign, a traffic light, an oncoming vehicle, a pedestrian, as well as lane markings and text on the road. The image recognition processor 22 detects each of these objects and also determines the distance from the object to the vehicle 1 from the parallax between the main camera 5a and the auxiliary camera 5b in order to generate a distance image.

[0025] In particular, as in Fig. Figure 6 shows the image recognition processor 7 taking a portion of the current image area Im, captured by the onboard camera 5, as the effective image area le. Then, as shown in Fig. Figure 7A shows that, based on an image of the effective image area le, which is loaded by cameras 5a and 5b, a distance image is generated which is formed from a distance-pixel distributed image which has distance information, as shown in Fig. Figure 7B shows the following. With respect to the vertical direction of the effective image area le in the captured image, the bottom side is the near side and the top side is the far side.

[0026] The camera control unit 23 reads an average brightness value of the image data processed by the image recognition processor 22, sets an exposure time (i.e. shutter speed) which provides optimal brightness in an image acquisition process of a subsequent single image, and controls the exposure of each of the cameras 5a and 5b on the basis of this exposure time.

[0027] The image mode switching unit 24 switches between a normal mode, in which the image recognition area is set to the entire effective image area le, and an area limitation mode, in which the effective image area le is limited, depending on whether an activation command signal is output by the airbag activation command unit 13. If, in this area limitation mode, an image of the airbag 7a appears in the effective image area le, as shown in Fig. As shown in Figure 7A, the boundary is detected with the front image, and a captured vertical image area on the near side relative to this boundary is masked as an unusable area A1. Consequently, a captured vertical image area on the far side relative to the boundary serves as a usable area A2, and this usable area A2 is set as the image recognition area.

[0028] The object recognition processor 25 performs a known grouping process on the distance image (see Fig. 7B) of the image recognition area, which is set by the image mode switching unit 24, and compares the distance image with a three-dimensional frame (window) that is pre-stored to detect objects in front of the vehicle 1, such as a vehicle ahead, a pedestrian, and a lane, as well as the distance to each object. The object recognition processor 25 then causes a memory unit, such as RAM, to temporarily store the detected objects and distances as object information. Since the method for determining the distance between each object in the vehicle 1 is generally known, a description thereof is omitted.

[0029] Based on the object information recognized by the object recognition processor 25 and temporarily stored in the memory unit, various types of driver assistance control, such as automatic steering control, automatic following control and automatic braking control, are executed to assist the driver while driving.

[0030] The image mode switching process described above, which is executed by the image mode switching unit 24, will now be described in more detail according to a Fig. The image mode switching process routine shown in section 3 is described.

[0031] In particular, this routine first checks in step S1 whether an activation command signal is received, which is issued by the airbag activation command unit 13 of the pedestrian protection airbag unit 11 to the airbag module 7. If the activation command signal is not received, the process proceeds to step S2 to execute normal mode and then exits the routine. If, on the other hand, the activation command signal is received, the process proceeds to step S3 to execute area limitation mode and then exits the routine. In normal mode, executed in step S2, the entire effective image area le, which is in Fig. 6 is shown as the image recognition area is set.

[0032] The area limitation mode described above in step S3 is implemented according to a Fig. The subroutine depicted in section 4, the range limitation mode, is executed. In this subroutine, the usable range A2 is first defined in step S11 (see Fig. 7) set by detecting the boundary between the unusable area A1 and the usable area A2.

[0033] Methods for detecting the boundary between the non-usable area A1 and the usable area A2 include a first method in which detection is based on the color of the airbag 7a, and a second method in which detection is based on the number of effective distance pixels in the distance image. In the first method, in which detection is based on the color of the airbag 7a, the color of the airbag 7a (which is set, for example, on the basis of a combination of brightness values ​​(R, G, and B) of 256 grayscale levels) is pre-stored in a memory unit such as the ROM, and an area with the brightness value corresponding to the airbag 7a is detected from the current image of the effective image area le, which is captured by the main camera 5a, as shown in Fig. 7A is displayed. Then the upper edge 7d, which has the detected brightness value, is located and set as the boundary between airbag 7a and the driving environment in front of vehicle 1. A captured vertical image area on the near side relative to this boundary is masked as the unusable area A1. Consequently, the captured vertical image area on the far side relative to the boundary is set as the usable area A2.

[0034] The process then proceeds to step S12 to set the usable area A1 as the image recognition area, and then exits the routine.

[0035] In a case where the pedestrian protection mechanism is of the hood-raising type, which protects a pedestrian by raising the rear of the front hood 4, the color from the rear edge of the front hood 4 can be stored, and an area corresponding to a brightness value from the current image is detected, so that its upper edge is set as the boundary.

[0036] According to this method, the boundary with the unusable area A1, caused by the appearance of airbag 7a, is determined from the current image and the distance image, thus ensuring a maximum usable area A2. In the second method, where detection is based on the number of effective distance pixels in the distance image, if airbag 7a appears in the effective image area le, seams and folds in airbag 7a would be erroneously detected as effective distance pixels. This causes the number of effective distance pixels (i.e., the number of pixels actually used for calculating the distance) to decrease significantly. Therefore, the number of effective distance pixels is determined for each row (in the horizontal direction), and the number of effective distance pixels in adjacent rows from row j1 to row j16 in the vertical direction is... Fig. 7B compared. A series in which the number of effective spacing pixels increases drastically (i.e., in the j7 series in Fig. 7C) is identified as the boundary between the airbag 7a and the driving environment in front of the vehicle 1, and a captured vertical image area on the near side relative to this boundary is masked as the unusable area A1. As a result, a captured vertical image area on the far side relative to the boundary is set as the usable area A2.

[0037] Alternatively, the number of effective spacing pixels in the horizontal direction is combined with a preset number of unusable area determination pixels for each of the rows from the j1 row to the j16 row in the vertical direction. Fig. 7B compared, and the position where one condition (i.e., the number of effective pixels ≥ the unusable determination value) changes to another condition (i.e., the number of effective pixels < the unusable determination value) is considered the boundary (i.e., between the j6 row and the j7 row in Fig. 7C) identified. A captured vertical image area on the near side relative to the boundary is masked as the non-usable area A1, and a captured vertical image area on the far side relative to the boundary is set as the usable area A2.

[0038] In the case where the pedestrian protection mechanism is of the hood-lift type, although protrusions and indentations on the inner surface of the hood would be erroneously detected as the number of effective distance pixels, this number of effective distance pixels is an extremely small value. Therefore, the boundary is identified based on this number of effective distance pixels, similar to the description above.

[0039] Fig.Figure 5 shows an area limitation mode subroutine according to a different procedure. In the subroutine described above, the boundary between the airbag 7a and the driving environment in front of the vehicle 1 is identified based on either the current image or the distance image. Alternatively, in this example, the non-usable area A1 to be masked and the usable area A2 are preset, and the image recognition area is uniquely set based on the preset non-usable area A1 and usable area A2.

[0040] In this subroutine, the pre-stored unusable area A1 and usable area A2 are first read in step S21. These areas A1 and A2 are determined from tests or simulations and are stored as fixed data in a memory unit, such as the ROM. Specifically, according to the test or simulation, the boundary of the upper edge 7d of the airbag 7a is detected from the appearance of the effective image area le when the airbag 7a is inflated and deployed. Then, a captured vertical image area on the near side relative to this boundary is masked as the unusable area A1, and a captured vertical image area on the far side relative to the boundary is set as the usable area A2.

[0041] The process then proceeds to step S22, where an image area corresponding to the unusable area A1 of the effective image area le is masked, while an image area corresponding to the usable area A2 is set as the image recognition area. The process then exits the routine.

[0042] Since the usable area A2 is set immediately according to this procedure, it is possible to quickly cope with the inflation and deployment of the airbag 7a.

[0043] Even if, as in this example, the pedestrian protection airbag 7a inflates and deploys and appears within the field of view of the onboard camera, the area of ​​the image containing the airbag 7a is masked as the unusable area A1, and the vertical area of ​​the image is set as the usable area A2. Since this usable area A2 is set as the image recognition area, the driver assistance control can resume based on the image captured by the camera without restricting the operation of the pedestrian protection mechanism. Consequently, in addition to effectively protecting a pedestrian from a secondary collision in the event of a collision, a secondary collision involving the vehicle can also be prevented.

[0044] The present invention is not limited to the example described above. For example, the present invention is also applicable to a type in which the pedestrian protection mechanism causes the rear of the front hood 4 to spring upwards to act as a damping element. In this case, the rear end of the front hood 4, which appears in the image, is defined as the boundary between the non-usable area A1 and the usable area A2.

[0045] An image processing device includes an onboard camera, an image recognition processor, an object recognition processor, and an image mode switching unit. The onboard camera captures an image in front of a vehicle. The image recognition processor generates a distance image within an image recognition area based on the captured image. The object recognition processor detects an object within the image recognition area and its distance based on two image types. The image mode switching unit toggles the image recognition area to be detected by the object recognition processor. The image mode switching unit checks whether a collision with a pedestrian is detected and issues an activation command signal to a pedestrian protection mechanism.When an output of the activation command signal is detected, the image mode switching unit switches to a mode in which an image area without a pedestrian protection mechanism is set as the image recognition area.

Claims

[1] Image processing device (21) comprising: an on-board camera (5) that takes a picture of a driving environment in front of a vehicle (1); an image recognition processor (22) that generates a distance image within an image recognition area based on the image captured by the onboard camera (5); an object recognition processor (25) that recognizes an object within the image recognition area and a distance to the object based on the image captured by the onboard camera (5) and the distance image generated by the image recognition processor (22); and an image mode switching unit (24) that switches the image recognition area to be recognized by the object recognition processor (25), wherein The image mode switching unit (24) checks whether a collision with a pedestrian is detected and an activation command signal is issued to a pedestrian protection mechanism, and In a case where an output of the activation command signal is detected, the image mode switching unit (24) switches from a normal mode in which a normal effective image area (Ie) is set as the image recognition area to an area limitation mode in which an image area in which the pedestrian protection mechanism does not appear is set as the image recognition area. [2] Image processing device (21) according to claim 1, wherein the area limitation mode comprises: identifying the pedestrian protection mechanism that appears in the effective image area (Ie) based on a color of the pedestrian protection mechanism, detecting a boundary where the color of the pedestrian protection mechanism does not appear, and setting a captured vertical image area (A2) at a far side of the boundary as the image recognition area. [3] Image processing device (21) according to claim 1, wherein the area limitation mode comprises: determining a number of effective distance pixels in the horizontal direction of the effective image area (Ie) for each of rows in a vertical direction, setting a row in which the number of effective distance pixels is greater than a preset number of non-usable area determination pixels as the boundary, and setting the captured vertical image area (A2) at a far side from the boundary as the image recognition area. [4] Image processing device (21) according to claim 1, wherein an area in which the pedestrian protection mechanism appears in the effective image area (Ie) is provisionally set as a non-usable area (A1), and wherein the area limitation mode includes: setting a captured vertical image area (A2) at a far side of a boundary of the non-usable area (A1) as the image recognition area.

Images