Camera control device and vehicle control device

Abstract

Provided are a camera control device and a vehicle control device that can be realized by turning headlights off for a short time and comparing images when light is turned off and turned on in order to detect a reflective object using a camera and use the reflective object for object detection in driving assistance, and that, due to the camera and the headlights being physically separated, make it possible to asynchronously control the imaging timing of the camera and the turning-on / -off of the headlights as well as to capture images near the timing of switching between turning on and turning off. The time phase of an imaging timing pattern (imaging phase) of a camera imaging unit 2 is adjusted, as a result of which an imaging time interval at the time of switching between turn-off and turn-on of a headlight unit 10 is curtailed (shortened).

Description

[Technical Field]

[0001] The present invention relates to a camera control device and a vehicle control device that utilizes headlight control by the vehicle's own camera at night in a vehicle equipped with a camera, headlights, and an Advanced Driver-Assistance Systems (ADAS) function, to improve recognition accuracy at night and provide appropriate driving assistance using that recognition. [Background technology]

[0002] When driving at night, visibility is low due to the darkness of the surroundings, and drivers may be slower to notice dangers, such as pedestrians or bicycles without lights, than during the day. If reflective objects are attached to pedestrians or bicycles, the headlights of the vehicle will reflect off the reflective objects, which will help drivers to avoid delays in noticing dangers.

[0003] 2. Description of the Related Art A method has been proposed for assisting drivers in nighttime driving using a vehicle system that has a camera and controls headlight illumination using information from the camera.

[0004] Light distribution control is a technology that has been proposed for a long time. For example, as in Patent Document 1, the distance between the vehicle and the preceding vehicle is measured and high beams and low beams are switched depending on the distance at night. Various proposals have been made as conditions for switching headlight control, such as the presence or absence of not only a preceding vehicle but also an oncoming vehicle.

[0005] In order to determine whether an object is a reflective or luminous object, the headlights of the vehicle are turned off for a short period of time. Objects that disappear when the headlights of the vehicle are turned off are reflective objects, while objects that do not disappear are luminous objects. By turning off the headlights, it is possible to distinguish between reflective and luminous objects.

[0006] Patent Document 2 proposes a technology that removes reflective objects and makes it easier to detect the lights of oncoming vehicles. The headlights and camera are controlled in synchronization at the same time, and the camera captures an image only at the moment the headlights are turned off, thereby removing reflective objects. As a result, the image captured by the camera contains only luminous objects, making it easier to detect the lights of oncoming vehicles.

[0007] Next, in order to detect reflective objects and light-emitting objects, Patent Document 3 synchronizes the headlights and camera at the same time and captures images alternately when the headlights are turned off and on, thereby achieving detection of reflective objects and light-emitting objects. Furthermore, by dividing the headlight illumination area and controlling the lights independently for each area, it is possible to extend the off period for only specific areas to prevent glare, for example. Meanwhile, since the image capture timing is the timing when the lights are turned off and on for the entire area, the longer the off period for a specific area, the longer the interval between images capture.

[0008] Thus, it has been proposed to combine headlight control with the removal or detection of reflective objects. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Publication No. 6-84099 [Patent Document 2] Japanese Patent Application Laid-Open No. 2007-76429 [Patent Document 3] Japanese Patent Application Laid-Open No. 2011-110999 Summary of the Invention [Problem to be solved by the invention]

[0010] When detecting reflective objects by controlling the illumination of headlights, it is desirable to keep the interval between imaging timing when the headlights are on and off (imaging time interval) as short as possible. When comparing images when the headlights are on and when they are off, if the time interval between image acquisitions is long, the object captured in the image may move a lot and disappear from the screen, making it necessary to determine whether it disappeared due to a reflective object or because the object moved.

[0011] If the time interval between the imaging timing of the lights on and off is shortened, it becomes difficult to synchronize and control the camera imaging timing and headlight illumination control at the same time. If the camera and headlights are independent devices located physically apart and information between the devices is communicated via CAN, delays of several tens of milliseconds or more are required due to processing by the microcontrollers in each device and communication delays between devices via CAN, making synchronous control difficult. For this reason, it is necessary to control the camera imaging timing and headlight illumination control asynchronously. In both Patent Document 2 and Patent Document 3, the camera imaging timing and headlight illumination control are synchronized, making them unsuitable for imaging with short time intervals between lighting and turning off the lights.

[0012] Furthermore, in order to use the camera not only for light distribution control but also for object detection used in driving assistance such as automatic emergency braking, it is necessary to repeatedly capture images within a specific cycle, for example, within a 50 ms cycle. Taking images only when the lights are turned off, as in Patent Document 2, or extending the light-off period and increasing the imaging interval to prevent dazzle, as in Patent Document 3, are not suitable for capturing images for driving assistance.

[0013] Therefore, in order to detect reflective objects using a camera and use it for object detection in driving assistance, the challenge is to shorten the timing of capturing images when the lights are turned on and off, and to provide a means to control the camera's capture and the on / off of the headlights asynchronously.

[0014] Therefore, the present invention aims to provide a camera control device and a vehicle control device that have means for solving this problem.In order to shorten the imaging timing when the lights are on and off and to control the camera imaging and the on / off of the headlights asynchronously, the camera control device has three means: (1) a variable timing control means that varies the timing of the camera imaging, (2) an illumination control means that controls the on / off of the headlights using a duty ratio, and (3) a control means that performs phase control to align the timing phases of the imaging timing to shorten the imaging timing when the lights are on and off and the headlight illumination control timing, since the timing is asynchronous.

[0015] Furthermore, the present invention aims to utilize the above three means to detect reflective objects such as pedestrians and bicycles, and (4) to provide a vehicle control device that has a means for providing driving assistance such as automatic emergency braking using the detection information of reflective objects. [Means for solving the problem]

[0016] In order to achieve the above object, the camera control device according to the present invention is a camera control device that controls a headlight unit that illuminates the area ahead of the vehicle and a camera imaging unit that images the area ahead of the vehicle, wherein the camera control device sets a lighting / exit pattern of a predetermined frequency and duty for the headlight unit, and controls the headlight unit to turn on and off using the lighting / exit pattern, and sets an imaging timing pattern for the camera imaging unit that is different from the lighting / exit pattern, based on the predetermined frequency and duty for the headlight unit, and has a first periodic imaging time interval and a second imaging time interval that is shorter than the first imaging time interval, and adjusts the time phase of the imaging timing pattern of the camera imaging unit to reduce the imaging time interval when the headlight unit switches between being turned on and off.

[0017] Furthermore, when the reflective object determined to be a stationary object is located at a certain distance, the vehicle control device determines that the reflective object is a road boundary and issues an alarm or controls the steering of the host vehicle to prevent intrusion into the road boundary. Furthermore, when the reflective object is a moving object, the vehicle control device determines that the moving object is a pedestrian or a bicycle based on a comparison of the relative speed with the host vehicle, predicts a future trajectory from a past trajectory, calculates a risk of a collision with the host vehicle, and issues an alarm or automatic emergency brake control when it is determined that the risk of a collision with the host vehicle is high. [Effects of the Invention]

[0018] As described above, according to the present invention, by periodically turning off the headlights for a short period of time and capturing an image at a short timing at the boundary between the off and on states, it is possible to detect reflective objects and luminous objects. Of the reflective objects, pseudo objects that appear as objects on the road due to reflections of headlights, etc., can be correctly removed, thereby avoiding erroneous headlight light distribution control or automatic emergency braking due to pseudo objects. Furthermore, by using reflective objects such as curbs and guardrails as road boundaries and using them as a substitute when lane detection is not possible, the performance of automatic departure avoidance systems can be improved. This makes it possible to realize various advanced driver assistance systems (ADAS).

[0019] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a system configuration diagram according to a first embodiment of the present invention. [Figure 2] 10A and 10B are examples of screen displays of reflective objects and luminous objects on a curve in the first embodiment of the present invention, where (a) is an example of an image when the lights are on at time t1, (b) is an example of an image when the lights are off at time t1, and (c) is an example of an image when the lights are off at time t2. [Figure 3]1A and 1B are examples of movement vectors in an image when the light is on in the first embodiment of the present invention, where (a) is an example of an image when the light is on at time t1, and (b) is an example of an image when the light is on at time t2. [Figure 4] 1A is a timing chart of lighting and imaging in the first embodiment of the present invention, where FIG. 1A is a timing chart before phase adjustment of the timing of turning on and off the headlights and imaging, and FIG. 1B is a timing chart after phase adjustment of the timing of turning on and off the headlights and imaging. [Figure 5] 10 is a flowchart showing phase adjustment of light on / off and image capture timing in the first embodiment of the present invention. [Figure 6] 4 is a flowchart of object recognition in the first embodiment of the present invention. [Figure 7] 4 is a flowchart of lane and road boundary recognition in the first embodiment of the present invention. [Figure 8] 4 is a flowchart of light distribution control in the first embodiment of the present invention. [Figure 9] 3 is a flowchart of automatic emergency brake control in the first embodiment of the present invention. [Figure 10] 3 is a flowchart of an automotive lane departure control according to the first embodiment of the present invention. [Figure 11] 10 is a timing chart of lighting and imaging in a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0022] [First Example] In a first embodiment of the present invention, in order to detect a reflective object with a camera and use it for object detection in driving assistance, the headlights are turned off for a short period of time, and the timing for capturing images when the headlights are turned on and off is shortened, and the camera image capturing and the turning on and off of the headlights are controlled asynchronously. To address this issue, the following are provided: (1) a variable timing control means for varying the timing for capturing images with the camera; (2) an illumination control means using a duty ratio as a means for controlling the turning on and off of the headlights; (3) a phase control means for shifting the phase of the imaging timing in order to adjust the imaging timing and the headlight illumination control timing in order to shorten the timing for capturing images when the headlights are turned on and off; and (4) a means for providing driving assistance such as automatic emergency braking using the detection information of the reflective object.

[0023] Prior to describing the configuration of the vehicle system of the first embodiment, the principle of the first embodiment will be described with reference to FIGS.

[0024] Figure 2 shows an example of the display of reflective and light-emitting objects on a curve. The vehicle is approaching a curve. Figure 2(a) shows an image at time t1 when headlights are illuminated. The light-emitting objects are headlights 100 of an oncoming vehicle. The reflective objects include reflective curbstones 101 and guardrails 102, which are spaced at regular intervals. When the visibility of road boundaries is low, the approximate location of the road boundaries can be determined by using these spaced reflective objects. In addition, there is a reflective object 103 that reflects when the vehicle's headlights are turned on. This reflective object 103 may be a reflection from a person or bicycle, but it may also be a pseudo object that appears as an object on the road due to the reflection of the headlights, etc. Figure 2(b) shows an image at the same time t1 when the headlights are off. No reflective objects are displayed on the screen, and only the light-emitting objects, headlights 100 of an oncoming vehicle, are displayed on the screen. By comparing the lit image (FIG. 2(a)) with the unlit image (FIG. 2(b)), it is possible to separate not only curbs and guardrails but also other reflective objects 103. FIG. 2(c) shows an image taken at time t2, after time t1, when the headlights are off. At this time, the host vehicle approaches the curve and the oncoming vehicle turns from the curve towards the host vehicle for the duration of time t2-t1, so the headlights 104 of the oncoming vehicle are positioned at the bottom of the screen compared to FIG. 2(b). Furthermore, as time passes, the oncoming vehicle moves outside the screen, and the light-emitting objects, the headlights of the oncoming vehicle, disappear from the screen.

[0025] Since there is a time difference between when the headlights are turned on and when they are turned off, Figures 2(a) and (b) cannot occur at the same time. Time t1 is the image when the headlights are on, and if the time when the headlights are turned off is a different time, t2, which is later than t1, the image when the headlights are off will be as shown in Figure 2(c). If the interval between t2 and t1 is long, the environmental conditions will change significantly, increasing the possibility that the light-emitting object will disappear from the screen, so when comparing images with the lights on and off, it can be said that a short time interval is preferable.

[0026] In Figure 2, objects that are not detected in the image when the light is off can be identified as reflective objects, while Figure 3 shows how to distinguish between moving objects, stationary objects, and pseudo objects among reflective objects.

[0027] Figure 3 shows examples of detecting moving, stationary, and pseudo-object reflective objects. A moving reflective object is a bicycle with a reflective object attached somewhere on the bicycle, such as the rear wheel. A stationary reflective object is a reflective object attached to a utility pole or similar. A pseudo-object is an object that appears to be an object but does not actually exist, such as a reflection of the vehicle itself or the reflection of the vehicle's headlights on the road.

[0028] At time t1, the headlights are turned on, and a reflective object 111 and pseudo reflective object 113 on the inside of the roadway, and a reflective object 112 on the outside of the roadway are detected (Fig. 3(a)). At time t2, after time t1, the headlights remain on, and a reflective object 114 and pseudo reflective object 116 on the inside of the roadway, and a reflective object 115 on the outside of the roadway are detected (Fig. 3(b)). By comparing the images at time t1 and time t2, the vector of the movement direction can be calculated from the position of the reflective object. The movement vector of the reflective object on the roadway is 117, and the movement vector of the reflective object outside the roadway is 118. Because the pseudo reflective object 116 is always in the same place on the screen, its movement vector 119 is zero (Fig. 3(b)).

[0029] This allows pseudo objects to be distinguished by the absence of a movement vector, and stationary and moving objects to be distinguished by the direction and magnitude of the movement vector.

[0030] FIG. 4 shows the principle of imaging at variable timing when the headlights are on and off in the first embodiment of the present invention.

[0031] FIG. 4(a) shows a timing chart of the headlight on / off and imaging timing before phase adjustment, and FIG. 4(b) shows a timing chart after phase adjustment.

[0032] First, the camera issues a duty cycle command to the headlights to turn them on and off. The headlights receive the duty cycle command and repeat turning them on and off at that duty cycle. In Figure 4, the duty ratio for turning the lights on and off is 1:9, so for every 1 minute of off time, the lights are turned on and off repeatedly at intervals of 9 minutes.

[0033] The camera has a shutter control that allows variable imaging timing, and captures images at short and long time intervals. The time intervals of the camera's shutter control are repeated periodically and asynchronously, independent of the headlights. In Figure 4(a), the time intervals between times t1 and t2 and between t2 and t3 are short, while the time intervals between t3 and t4 and between t4 and t5 are long. If an image is captured with the headlights off, a dark image is obtained, and if an image is captured with the headlights on, a bright image is obtained. In Figure 4(a), images taken at times t4, t9, and t14 are taken when the headlights are off, and images taken at other times are taken when the headlights are on.

[0034] In order to shorten the time interval between when the headlights are turned off and when they are turned on in order to compare images to determine whether there is a reflective object, and to shorten the time interval when the headlights are turned off in order to acquire multiple images when the headlights are off, it is necessary to synchronize the areas with short image capture intervals with the timing of headlight extinguishing.Since the turning on and off of the headlights and the image capture timing are both cyclical processes, by adjusting the phase of the image capture timing, it is possible to synchronize the areas with short image capture intervals with the timing of headlight extinguishing.

[0035] Figure 4(b) shows a timing chart after adjusting the phase of the headlight on / off and imaging timing. Images taken at times t1, t2, t6, t7, t11, and t12 are images taken when the headlights are off, while images taken at other times are images taken when the headlights are on. Whereas Figure 4(a) only captures one image when the headlights are off, Figure 4(b) allows for two images to be taken when the headlights are off. It is also possible to capture images near the transition between turning the headlights on and off. In this way, by adjusting the phase of the headlight on / off and imaging timing, which are asynchronous but cyclical, it is possible to capture many images when the headlights are off and near the transition between turning on and off, and to capture images at regular intervals when the headlights are on.

[0036] That is, if the headlight on / off duty, the corresponding variable imaging timing of the camera, and the number of images acquired when the lights are off and on are determined in advance as, for example, 2 images when the lights are off and 6 images when the lights are on, then in Figure 4(a) there is 1 image when the lights are off and 7 images when the lights are on, and since the numbers are different, by adjusting the phase of the imaging timing, in Figure 4(b) there are 2 images when the lights are off and 6 images when the lights are on, which is the same as the determined number of images when the lights are off and 6 images when the lights are on, and so this is the state after phase adjustment. In this way, self-adjustment (self-test) of the phase adjustment can be performed.

[0037] An example configuration of a vehicle system according to a first embodiment of the present invention, in which a camera is used to detect reflecting objects and to detect objects for driving assistance, is shown in Fig. 1. The camera is not particularly limited to a monocular camera or a stereo camera, but in the first embodiment, a stereo camera is used which has two lenses, one on the left and one on the right, and can detect the distance to an object by trigonometry.

[0038] Vehicle 1 is composed of a camera imaging unit 2 that has a built-in camera lens and image sensor and captures images of the area in front of the vehicle, a camera control unit 3 that processes the captured images and detects and recognizes objects, a vehicle control unit 4 that uses the recognition information to provide driving assistance such as brake control and steering control, an illuminance meter 5 that indicates the illuminance outside the vehicle, a CAN 6 that performs intra-vehicle communication, a headlight unit 10 that controls the headlights that illuminate the area in front of the vehicle, a map unit 11 that has information on the vehicle's position and its surrounding environment, and a steering unit 12 and a brake unit 13 that are vehicle actuators.

[0039] The camera imaging unit 2 has the function of capturing images, capturing images, and outputting the RAW images via an LVDS cable. Although not shown in the figure, it has left and right lenses, an image sensor, a serializer that converts digital image data into a serial signal, and a shutter setting unit 27. The shutter setting unit 27 stores timing information for the imaging shutter. For example, it stores information (imaging timing pattern) such as a period of 350 ms, a total of 8 shutters, with 2 shutters at 25 ms and 6 shutters at 50 ms (see Figure 4).

[0040] The camera control unit 3 is connected to the camera imaging unit 2 via an LVDS cable and is composed of an image processing unit 20 that processes the transferred RAW images, a recognition unit 21 that recognizes objects and lanes in the image, a light control unit 23 that issues instructions for light distribution control to switch between high beam and low beam depending on the timing of turning on and off the headlights and the presence or absence of vehicles, and a CAN-IF unit 22 that is connected to the CAN bus and sends and receives data.

[0041] The image processing unit 20 has a function to issue a shutter instruction, which is the image capture timing, to the camera image capture unit 2, and a function to generate an image required for recognition from the transferred RAW image. The former is performed by an image capture timing calculation unit 30 that calculates the image capture timing of the camera image capture unit 2, and an image capture phase adjustment unit 31 that shifts the phase of the image capture timing to coincide with the on / off of the light and issues a shutter instruction, which is the image capture timing, to the camera image capture unit 2. Although not shown in the figure, the latter is composed of a deserializer that generates RAW image data from a serial signal, a parallax image generation unit 33 that generates a parallax image having distance information in pixel units from the left and right RAW images, a luminance image generation unit 32 that generates a luminance image from one of the left and right RAW images, an image comparison unit 34 that compares two or more images with a time difference, and an edge image generation unit 35 that generates an edge image from the gradient of the luminance image.

[0042] The recognition unit 21 is composed of a light-off duty calculation unit 40 that determines the duty of the headlights on and off, a pseudo-object removal unit 41 that removes pseudo-objects that do not actually exist, such as reflections of the vehicle's lights, a moving object determination unit 42 that determines whether an object is stationary or moving, a lane / road boundary detection unit 43 that detects lanes from the image and detects road boundaries using reflective objects such as the lane, curbs, road shoulders, and guardrails, and a light distribution detection unit 44 that detects headlights and taillights of other vehicles for light distribution control.

[0043] The light control unit 23 has a duty instruction unit 70 for determining the timing of turning on and off the headlights using a duty ratio, and a light distribution instruction unit 71 for instructing light distribution control to switch between high beam and low beam depending on the presence or absence of a vehicle.

[0044] The vehicle control unit 4 is composed of a lane departure control unit 24 that determines whether the vehicle will deviate from its lane and, if so, issues an alarm or steering control command to return the vehicle to its lane; an automatic emergency braking control unit 25 that determines whether the vehicle will collide with an obstacle and automatically issues an emergency braking command when there is a high possibility of a collision; and a CAN-IF unit 26 that connects to the CAN bus and sends and receives data.

[0045] The lane departure control unit 24 is composed of a vehicle trajectory calculation unit 52 that predicts and calculates the vehicle's trajectory, a departure determination unit 53 that determines whether the vehicle will deviate from a road boundary based on the vehicle's trajectory and road boundary information, and a control unit 51 that issues an alarm or steering control instructions when the vehicle deviates from a road boundary.

[0046] The automatic emergency braking control unit 25 is composed of an obstacle trajectory calculation unit 60 that calculates the trajectory of an obstacle, a risk calculation unit 61 that calculates the trajectory of the vehicle and calculates the risk of collision with an obstacle, and a control unit 62 that generates a warning or braking instruction based on the collision risk.

[0047] In this embodiment, the turning on and off of the light and the image capturing by the camera are performed independently and asynchronously, but both are cyclical processes, and by adjusting the phase, images are captured a specific number of times when the light is on and when it is off. Although the example shows the light being turned on automatically, the invention is not limited to this and can also be applied to manual lighting by a human being.

[0048] Figure 5 shows a flowchart for adjusting the phase of the light on / off and imaging timing.

[0049] First, when controlling the turning on and off of the light, setting begins when the illuminance meter 5 indicates a decrease in brightness. The illuminance is calculated by the illuminance meter 5 and transmitted to the camera control unit 3 (S11). If the illuminance is not below the threshold value (S12), the turning on and off control is not started; if the illuminance is below the threshold value (S12), the turning on and off control is started. The light-off duty calculation unit 40 calculates the light-off duty using data stored in memory (S13). The light-off duty indicates the cycle time (the time interval between turning on and off) and the ratio of the time between turning on and off. Using the light-off duty, the image processing unit 20 and the light control unit 23 each perform the following. In the image processing unit 20, the imaging timing calculation unit 30 uses the light-off duty to calculate the shutter timing for imaging, i.e., the number of times to capture images when the light is off and on, and instructs the camera imaging unit 2 of the shutter timing (S14). The camera 200 calculates shutter timing so that many images are captured in a short period of time when the headlights are off, and images are captured at regular intervals (periodically) when the headlights are on, and captures images around the time the headlights switch between on and off. Specifically, the image capture timing calculation unit 30 sets, in the camera 2, an image capture timing pattern that is different from the on / off pattern, with regular intervals and intervals shorter than the regular intervals, based on a predetermined frequency and duty for the headlight unit 10 (described later). The camera 20 sets shutter timing in the shutter setting unit 27 based on the shutter timing (image capture timing pattern) transferred from the image processing unit 20, and captures images at the set timing (S15). In parallel with the image processing unit 20, the light control unit 23 receives the off-duty and instructs the headlight unit 10 to set the off-duty (S16) via the duty instruction unit 70. Specifically, the duty instruction unit 70 sets, in the headlight unit 10, an on / off pattern with a predetermined frequency (corresponding to the time interval between off and on) and duty. Furthermore, the light distribution instruction unit 71 controls (the light distribution of) the headlight unit 10 so as to turn on and off using the light on / off pattern. The headlight unit 10 sets a light off duty (light on / off pattern) in the light off duty setting unit 28, and periodically turns on and off the headlight unit 10 at the light off duty determined by the light distribution control unit 29 (S17).For example, if the cycle is 1 sec and the extinguishing duty is 1:9, the light will be turned on and off repeatedly with an extinguishing time of 100 ms and an on time of 900 ms (see FIG. 4(a)).

[0050] Next, the imaging phase adjustment unit 31 performs phase adjustment to adjust the phase of the headlights turning on and off and the imaging timing. The imaging phase adjustment unit 31 shifts the phase of the shutter timing (imaging timing pattern) and instructs the camera imaging unit 2 on the phase information (S18). The camera imaging unit 2 sets the phase information in the shutter setting unit 27 and performs imaging by shifting the phase to the set timing (S19). The imaging phase adjustment unit 31 measures the brightness of specific pixels in each screen (for example, pixels in locations that are not directly illuminated by the headlights but that always show differences depending on whether the headlights are on or off), and determines whether the image is an image with the headlights on or off based on the brightness. It then checks whether the periodic number of images with the headlights on and the number of images with the headlights off are as expected (i.e., whether the number of images with the headlights off and the headlights on are the same as the predetermined number) (S20). If the number of images with the headlights on and off differs (S21), the imaging phase adjustment unit 31 performs phase adjustment again to perform phase adjustment (S18). In this way, the phase adjustment has a self-adjustment (self-test) function.

[0051] After automatic phase adjustment is performed when the light is first turned on (see Figure 4(b)), recognition processing is performed on obstacles (target objects) on the screen.

[0052] A flowchart of object recognition is shown in Figure 6. The camera is assumed to be a stereo camera, but there is no particular limitation.

[0053] First, the image processing unit 20 of the camera control unit 3 receives left and right RAW images from the camera imaging unit 2 (S31). To extract the luminance of an object, the luminance image generation unit 32 generates a luminance image from the right RAW image (S32). Either the left or right RAW image may be used as the luminance image. The imaging phase adjustment unit 31 measures the luminance of specific pixels in each screen, for example, pixels in locations that are not directly illuminated by the headlights but that necessarily differ depending on whether the headlights are on or off. A predetermined luminance threshold is used to determine whether the headlights are on or off, and an on / off attribute indicating whether the headlights are on or off is calculated for the luminance image (S33). The parallax image generation unit 33 then uses trigonometry to generate parallax images with distances for each pixel from the left and right RAW images (S34). The parallax image generation unit 33 uses the parallax images to assign IDs to each detected object as identification numbers and to assign relative position attributes from the vehicle, including distance (S35).

[0054] Next, the image comparison unit 34 determines whether the object is a reflective object or a light-emitting object, and assigns a reflective / light-emitting object attribute. The attribute is updated as needed until it is confirmed. Objects that can be detected in a lit image may be both light-emitting and reflective objects, so they are given an indefinite attribute and are not confirmed. Objects that can be detected in an unlit image are confirmed as light-emitting objects. Since the light / unlit attribute is assigned to the image, image comparison is performed when one image taken at a certain time (t1) and the other taken at the time immediately before (t2) have the light-emitting attribute and the unlit attribute. The luminance images of the lit and unlit images are compared, and objects detected both when the light is unlit and when the light is on are confirmed (extracted) as light-emitting objects, and objects detected only when the light is on are confirmed (extracted) as reflective objects. The types of light-emitting and reflective objects are assigned as reflective / light-emitting object attributes (S36).

[0055] Furthermore, the image comparison unit 34 compares multiple brightness images taken at consecutive times, and calculates and adds a movement vector from the difference in the relative coordinates of the object (S37). Objects include both reflective and light-emitting objects. There are three types of comparisons of the brightness images of two images: lit images vs. lit images, unlit images vs. lit images, and lit vs. unlit images. When comparing lit images, movement vectors can be calculated for both luminous and reflective objects, but when comparing unlit images or lit vs. unlit images, movement vectors can only be calculated for light-emitting objects. Since the unlit portion of the reflective object cannot be detected by using multiple trajectory images taken at consecutive times, the movement vector is calculated by complementing and connecting the luminance images when lit. Each object retains as an attribute the movement vector between a certain time (t1) when it was imaged and the time immediately before (t2) when it was imaged. For a reflective object that is off at time (t1) and on at time (t2), the object is on at the time (t3) imaged immediately before time (t2), so the relative position at time (t1) is predicted using the relative positions at times (t2) and (t3). Because the location at time (t1) is unknown, it is assumed that the object moved at a constant speed in a straight line from times (t2) and (t3), and the relative coordinates at time (t1) are assumed, and a movement vector is calculated from the relative coordinates at times (t1) and (t2). Here, an example using time (t3) is shown, but to improve accuracy, multiple coordinates from image capture times going back further may be used, which will improve prediction accuracy.

[0056] Among the reflective objects, there are those that do not actually exist but appear to be objects due to the influence of the headlights of the vehicle. The pseudo object removal unit 41 can determine that objects with a reflective / light-emitting object attribute and a movement vector of zero are pseudo objects and remove them (S38). This allows pseudo objects that are not actually objects, such as reflections of the vehicle's lights, to be removed from obstacle targets in the light distribution control of the headlight unit 10 (which will be described later) (light distribution control that switches from high beam to low beam when an obstacle is detected). The moving object determination unit 42 then determines that objects with a vector of the same movement amount but in the opposite direction to that of the vehicle are stationary objects, and those with a different movement amount are moving objects, and adds the movement vector to the object information as a moving object attribute (S39).

[0057] In this way, the camera control unit 3 detects an object, determines whether the detected object is a reflective or luminous object by comparing the luminance images with the headlights on and off, and assigns a reflective / luminous object attribute to the object, assigns a relative position attribute including distance information from the vehicle using the parallax image, and further assigns a moving object attribute by comparing multiple luminance images.

[0058] Next, we will discuss lane recognition. In addition to normal lane detection, when the lane cannot be detected due to blurred lane markings, information on reflective objects is used to improve the lane detection rate.

[0059] FIG. 7 shows a flowchart of the lane / road boundary detection unit 43.

[0060] First, the brightness image generating unit 32 of the image processing unit 20 generates a brightness image (S41). The edge image generating unit 35 generates an edge image from the gradient of the brightness image (S42). The lane / road boundary detecting unit 43 detects lanes from the edge image and calculates them as coordinate points corresponding to a certain relative distance from the vehicle (S43).

[0061] In addition to the lanes, reflective objects placed at regular intervals on the shoulders of the road are also recognized as objects, and information about these objects is used. The lane and road boundary detection unit 43 extracts, from among the objects, reflective objects with reflective / light-emitting object attributes, stationary objects with moving object attributes, and multiple objects detected at equal intervals relative to the vehicle as a road boundary group (S44). That is, if the lane and road boundary detection unit 43 determines that reflective objects determined to be stationary objects are placed at equal intervals (at a constant distance) relative to the vehicle, it determines them to be road boundaries (groups). Depending on whether lanes are detected on the screen, it is selected whether to use lane detection information or road boundary groups as lane information. If lanes are detected on the screen (S45), the lane and road boundary detection unit 43 outputs the coordinate points of the detected lanes as lane information (S46). If no lane is detected on the screen (S45), the lane / road boundary detection unit 43 outputs a coordinate point that is a certain distance closer to the vehicle from the coordinates of the road boundary group as lane information (S47).

[0062] Next, a light distribution control method for controlling the high beam and low beam of headlights using information on recognized objects will be described. Light distribution control is a technology for switching between high beam and low beam depending on whether an object is present or not, i.e., using high beam when there is no object and low beam when there is an object. A feature of the first embodiment of the present invention is that reflective objects, including pseudo objects, are excluded from the targets of light distribution control using the reflective / light-emitting object attributes of the recognized objects (pseudo object removal unit 41).

[0063] Figure 8 shows a flowchart of light distribution control.

[0064] Since headlight light distribution control requires finely switching between high and low beams, which can be uncomfortable for the driver, objects are accumulated and saved from multiple images in order to improve the accuracy of obstacle detection (S51). The presence or absence of the same object in multiple images is checked (S52). If the same object is found, the reflective / light-emitting object attributes of the object are used, and if it is a reflective object, it is treated as if the object does not exist (S53). Furthermore, if it is not a reflective object (S53), it is determined whether the same object is a headlight or taillight of another vehicle detected by the light distribution light detection unit 44 (S54).

[0065] When three conditions (S52, S53, S54) are satisfied, it is determined that there is an effective object for which low beam should be used, and the light distribution instruction unit 71 generates a light distribution instruction to use low beam and sends the light distribution instruction to the headlight unit 10 (S55). The light distribution control unit 29 of the headlight unit 10 controls the headlight to use low beam (S56).

[0066] If any of the three conditions (S52, S53, S54) is not met, it is determined that there is no valid object for which low beam should be used, and control is performed to switch to high beam. A light distribution instruction for switching to high beam is generated by the light distribution instruction unit 71, and the light distribution instruction is sent to the headlight unit 10 (S57). The light distribution control unit 29 of the headlight unit 10 controls the headlight to switch to high beam (S58).

[0067] The reflective object determination condition (S53) and target object determination condition (S54) for light distribution control can be modified or expanded. For example, in this embodiment, the headlights and taillights of other vehicles are used as luminous objects, but if attributes of pedestrians and bicycles can be added, it is also possible to add the detection of reflective objects such as pedestrians and bicycles as conditions.

[0068] Next, using the object information and lane information recognized by the camera control unit 3, the vehicle control unit 4 calculates automatic emergency braking control and lane departure control.

[0069] Fig. 9 shows a flowchart of the automatic emergency braking control.

[0070] The camera control unit 3 transmits reflective / light-emitting object attributes, relative position attributes, and moving object attributes as recognized object information (S61). After receiving the recognized object information, the automatic emergency braking control unit 25 uses the accumulated object information to predict the trajectory of the obstacle in the obstacle trajectory calculation unit 60 (S62). For light-emitting objects with reflective / light-emitting object attributes, the obstacle trajectory calculation unit 60 predicts the trajectory using the relative position information in the object information. For reflective objects determined with reflective / light-emitting object attributes, since they cannot be detected during the off period and their relative position is unknown, the trajectory is predicted by predicting their position using linear interpolation or polynomial interpolation from relative positions taken multiple times back (S63). Since reflective objects correspond to both stationary and moving objects, moving object attributes are not used. However, moving object attributes may be used, and for stationary objects with moving object attributes, trajectory prediction may be skipped and the relative position of the object relative to the vehicle may be calculated based on the vehicle trajectory. For example, if the reflective object is a moving object, the obstacle trajectory calculation unit 60 determines that the moving object is a pedestrian or a bicycle based on the relative speed with respect to the host vehicle, and predicts its future trajectory from its past trajectory. Then, the risk calculation unit 61 calculates the risk of collision by determining whether the trajectories of the light-emitting object and the reflective object are on the host vehicle's path (S64). The control unit 62 issues an alarm or an emergency brake control command based on the collision risk calculated by the risk calculation unit 61 (when it is determined that the risk of collision with the host vehicle is high) (S65). Thereafter, the brake unit 13 of the actuator receives the emergency brake control command via the CAN 6, and performs brake control of the host vehicle based on the control command.

[0071] Figure 10 shows a flowchart of the automatic lane departure control.

[0072] The camera control unit 3 sends out reflective / light-emitting object attributes, relative position attributes, moving object attributes, and recognized lane information as information on the recognized object (S71). The vehicle trajectory calculation unit 52 predicts the trajectory of the vehicle from the steering information of the vehicle (S72). The deviation determination unit 53 inputs the trajectory predicted from the steering information and the recognized lane information and calculates the risk of deviation from the lane (S73). The control unit 51 issues an alarm or steering control instruction to avoid lane deviation (to prevent intrusion into the road boundary) based on the lane deviation risk from the deviation determination unit 53 (S74). Thereafter, the steering unit 12 of the actuator receives the steering control instruction via the CAN 6 and performs steering control of the vehicle based on the control instruction.

[0073] [Second Example] If the headlight on / off duty is synchronized with that of other vehicles and is the same duty, the headlights of other vehicles will always appear to be off from the driver's perspective. This can be avoided by using a multi-duty system with multiple duty cycles for the headlight on / off cycle.

[0074] The principle of the second embodiment is shown in Figure 11. Assume that there are two cycles, with the ratio of on to off in the first cycle being 1:9 and the ratio of on to off in the second cycle being 1:4. In other words, the headlights are periodically turned on and off multiple times with different duties (1:9 in the first cycle, 1:4 in the second cycle). The imaging timing of the camera is also variable to match the on / off duty of the two cycles. By performing phase adjustment, in the first cycle, two images are taken when the lights are off and six images are taken when the lights are on, and in the second cycle, three images are taken when the lights are off and five images are taken when the lights are on. This cycle is repeated. Two cycles is just one example; using a multi-duty system with three or four cycles significantly reduces the possibility of synchronization with other vehicles.

[0075] The system configuration diagram of the second embodiment may be the same as that shown in Fig. 1. The light-off duty calculation unit 40 is multi-duty, and accordingly, the duty instruction unit 70 of the light control unit 23 and the light-off duty setting unit 28 of the headlight unit 10 perform light control compatible with multi-duty. In addition, the imaging timing of the camera also performs variable imaging compatible with multi-duty in accordance with the headlights. Specifically, the imaging timing calculation unit 30, imaging phase adjustment unit 31 of the image processing unit 20, and shutter setting unit 27 of the camera imaging unit 2 perform variable shutter control compatible with multi-duty of on / off.

[0076] The first and second embodiments of the present invention can also be expanded as follows.

[0077] The headlight turn-off duty calculation by the recognition unit 21 can be set variably rather than fixedly by the turn-off duty calculation unit 40. To set the turn-off duty variably, it can be set based on a random number when the ignition is turned on, for example. Also, in order to reduce the frequency of turning off the headlights, it is possible to turn off the headlights only during specific periods or at specific locations based on the map information in the map unit 11.

[0078] [Summary of the first and second embodiments] As described above, the camera control device (camera control unit 3) of this embodiment is a camera control device that controls the headlight unit 10 that illuminates the area ahead of the vehicle and the camera imaging unit 2 that images the area ahead of the vehicle. The camera control device sets a lighting / exit pattern of a predetermined frequency and duty for the headlight unit 10 (duty instruction unit 70), and controls (the light distribution of) the headlight unit 10 to turn it on and off using the lighting / exit pattern (light distribution instruction unit 71). The camera control device sets an imaging timing pattern for the camera imaging unit 2 that is different from the lighting / exit pattern, with a periodic first imaging time interval and a second imaging time interval that is shorter than the first imaging time interval, based on the predetermined frequency and duty for the headlight unit 10 (imaging timing calculation unit 30), and adjusts the time phase (imaging phase) of the imaging timing pattern of the camera imaging unit 2, thereby reducing (shortening) the imaging time interval when the headlight unit 10 switches between being turned on and off (imaging phase adjustment unit 31).

[0079] The camera control device adjusts the time phase of the imaging timing pattern of the camera imaging unit 2 based on the number of images when the camera is turned on and the number of images when the camera is turned off by shifting the time phase (imaging phase adjustment unit 31).

[0080] The camera control device measures the brightness of a specific pixel in each image captured by the camera imaging unit 2 while shifting the time phase, determines whether the image is lit or off based on that brightness, and checks the number of periodic images when the light is on and the number of images when the light is off at the shutter timing in the time phase, thereby adjusting the time phase of the imaging timing pattern of the camera imaging unit 2 (imaging phase adjustment unit 31).

[0081] The camera control device compares a first luminance image and a second luminance image captured by the camera imaging unit 2 with the headlight unit 10 turned on and off to extract a reflective object, compares multiple luminance images to calculate the movement vector of the reflective object in the image, and determines the reflective object as a stationary object when the movement vector of the reflective object is opposite to the movement vector of the vehicle, determines the reflective object as a pseudo object when the movement vector of the reflective object is zero, and otherwise determines the reflective object as a moving object (luminance image generation unit 32, image comparison unit 34, pseudo object removal unit 41, moving object determination unit 42).

[0082] The camera control device variably sets the duty of the headlight unit 10 (light-off duty calculation unit 40).

[0083] The camera control device sets the duty of the headlight unit 10 based on a random number (light-off duty calculation unit 40).

[0084] The camera control device periodically turns on and off the headlight unit 10 a plurality of times with different duties (light-off duty calculation unit 40, duty instruction unit 70, light distribution instruction unit 71).

[0085] The camera control device turns off the headlight unit 10 only during a specific period or at a specific location based on map information (light-off duty calculation unit 40).

[0086] In the light distribution control of the headlight unit 10, which switches from high beam to low beam when an obstacle is detected, the camera control device excludes the pseudo object of the reflective object from the obstacle targets of the light distribution control (pseudo object removal unit 41, light distribution instruction unit 71).

[0087] In addition, the vehicle control device (vehicle control unit 4) of this embodiment is a vehicle control device that controls the vehicle based on information output from the camera control device, and when the reflective object determined to be the stationary object is located at a certain distance, the vehicle control device determines that it is a road boundary and issues an alarm or controls the steering of the vehicle to prevent intrusion into the road boundary (lane departure control unit 24).

[0088] Furthermore, the vehicle control device (vehicle control unit 4) of this embodiment is a vehicle control device that controls the host vehicle based on information output from the camera control device, and when the reflecting object is a moving object, the vehicle control device determines that the moving object is a pedestrian or a bicycle based on the relative speed with the host vehicle, predicts the future trajectory from the past trajectory, calculates the risk of a collision with the host vehicle, and issues an alarm or performs automatic emergency braking control when it is determined that the risk of a collision with the host vehicle is high (automatic emergency braking control unit 25).

[0089] As described above, according to this embodiment, by periodically turning off the headlights for a short period of time and capturing an image at a short timing at the boundary between the off and on states, it is possible to detect reflective objects and luminous objects, and by correctly removing pseudo objects that appear as objects on the road due to reflections of headlights, etc., it is possible to avoid erroneous headlight light distribution control and automatic emergency braking due to pseudo objects, and further, by using reflective objects such as curbs and guardrails as road boundaries and using them as substitutes when lane detection is not possible, it is possible to improve the performance of automatic departure avoidance systems, and various advanced driver assistance systems (ADAS) can be realized.

[0090] It should be noted that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0091] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a storage device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0092] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0093] 1. Vehicle 2. Camera imaging unit 3. Camera control unit (camera control device) 4. Vehicle control unit (vehicle control device) 5...Luminance meter 6···CAN 10 Headlight section 11. Map Section 12. Steering section 13 Brake section 20. Image processing section 21...Recognition section 22, 26 CAN-IF section 23 Light control unit 24 Lane Departure Control Unit 25 Automatic emergency braking control unit 27. Shutter setting section 28. Light-off duty setting section 29 Light distribution control section 30. Imaging timing calculation unit 31 Imaging phase adjustment unit 32. Luminance image generation unit 33 Parallax image generation unit 34. Image comparison section 35. Edge image generation unit 40. Lights-off duty calculation section 41...Pseudo object removal section 42...Moving object determination section 43 Lane and road boundary detection unit 44...Light distribution light detector 51, 62 Control section 52 Vehicle trajectory calculation unit 53 Deviation determination unit 60. Obstacle trajectory calculation unit 61 Risk Calculation Section 70 Duty indicator 71 Light distribution indicator 100...Headlights of an oncoming vehicle at time t1 101···Reflection from the curb reflective object at time t1 102···Reflection from the guardrail reflective object at time t1 103... Reflection from the headlights of the vehicle at time t1 (pseudo object) 104...Headlights of an oncoming vehicle at time t2 111···A reflective object of a bicycle in the roadway at time t1 112···Reflecting object of a stationary object outside the roadway at time t1 113···Pseudo object due to headlight reflection in the roadway at time t1 114···A reflective object of a bicycle in the roadway at time t2 115···Reflecting object of stationary object outside the roadway at time t2 116···Pseudo object due to headlight reflection in the roadway at time t2 117···Movement vector of the bicycle's reflected object in the roadway at time t2-t1 118···Reflection of a stationary object outside the roadway at time t2-t1Movement vector of the object 119···Movement vector of a pseudo object due to headlight reflection in the roadway at time t2-t1

Claims

1. A camera control device that controls a headlight unit that illuminates a front area of ​​a vehicle and a camera image capturing unit that captures an image of a front area of ​​the vehicle, The camera control device includes: setting a lighting pattern of a predetermined frequency and duty to the headlight unit, and controlling the headlight unit to turn on and off using the lighting pattern; a camera control device that sets an imaging timing pattern in the camera imaging unit, which is different from the on / off pattern, and which has a periodic first imaging time interval and a second imaging time interval that is shorter than the first imaging time interval, based on a predetermined frequency and duty for the headlight unit, and that adjusts the time phase of the imaging timing pattern of the camera imaging unit, thereby reducing the imaging time interval when switching between turning on and turning off the headlight unit.

2. 2. The camera control device according to claim 1, wherein the camera control device adjusts the time phase of the imaging timing pattern of the camera imaging unit based on the number of images when the camera imaging unit is turned on and the number of images when the camera imaging unit is turned off by shifting the time phase.

3. 2. The camera control device according to claim 1, wherein the camera control device measures the brightness of a specific pixel of each image captured by the camera imaging unit while shifting the time phase, determines whether the image is lit or extinguished based on the brightness, and checks the number of periodic images when the light is on and the number of images when the light is extinguished at the shutter timing in the time phase, thereby adjusting the time phase of the imaging timing pattern of the camera imaging unit.

4. 2. The camera control device according to claim 1, wherein the camera control device compares a first luminance image and a second luminance image captured by the camera imaging unit with the headlight unit turned on and off to extract a reflective object, compares a plurality of luminance images to calculate a movement vector of the reflective object in the image, and determines the reflective object as a stationary object when the movement vector of the reflective object is opposite to the movement vector of the host vehicle, determines the reflective object as a pseudo object when the movement vector of the reflective object is zero, and determines the reflective object as a moving object in any other cases.

5. 2. The camera control device according to claim 1, wherein the camera control device variably sets the duty of the headlight unit.

6. 6. The camera control device according to claim 5, wherein the camera control device sets the duty of the headlight unit based on a random number.

7. 2. The camera control device according to claim 1, wherein the camera control device periodically turns on and off the headlight unit a plurality of times with different duties.

8. The camera control device according to claim 1 , wherein the camera control device turns off the headlight unit only during a specific period or at a specific location based on map information.

9. 5. The camera control device according to claim 4, wherein the camera control device, in light distribution control of the headlight unit that switches from high beam to low beam when an obstacle is detected, excludes the pseudo object of the reflective object from obstacle targets of the light distribution control.

10. A vehicle control device that controls the host vehicle based on information output from the camera control device according to claim 4, The vehicle control device is characterized in that, when the reflective object determined to be a stationary object is located at a certain distance, it determines that it is a road boundary and issues an alarm or controls the steering of the vehicle to prevent intrusion into the road boundary.

11. A vehicle control device that controls the host vehicle based on information output from the camera control device according to claim 4, The vehicle control device, when the reflecting object is a moving object, determines that the moving object is a pedestrian or a bicycle based on a comparison of the relative speed with the host vehicle, predicts a future trajectory of the moving object from a past trajectory, calculates a risk of a collision with the host vehicle, and issues an alarm or automatic emergency braking control when it is determined that the risk of a collision with the host vehicle is high.

Images