Vehicle Assistance Systems

Abstract

The present invention relates to a vehicle assistance system. A method for assisting the operation of a vehicle traveling on a road includes: acquiring a visual image of the vehicle's surroundings with at least one visual camera having a field of view; and acquiring a thermal image of the vehicle's surroundings with at least one thermal camera having a field of view. Superimposing the thermal image on the visual image to produce a composite image. Detecting an object in the composite image. The vehicle assistance system adjusts at least one of the vehicle's driving direction and speed in response to detecting the object.

Description

Technical Field

[0001] The present invention relates generally to vehicle systems and, more particularly, to a vehicle assistance system having an imaging element that takes into account changes in visibility around the vehicle. Background Art

[0002] Current driver assistance systems (ADAS - Advanced Driver Assistance Systems) provide a range of monitoring functions in vehicles. In particular, ADAS can monitor the environment around the vehicle and inform the driver of the vehicle of conditions therein. To this end, ADAS can capture images of the surrounding environment and digitally process the images to extract information. This information is used to warn the driver of road obstacles located along the driving path. Common ADAS include automatic emergency braking, which is used to help prevent rear-end collisions, and lane detection, which is used to help keep the vehicle in the intended driving lane. Summary of the invention

[0003] In one aspect of the invention, a method for assisting operation of a vehicle traveling on a road includes: acquiring a visual image of the vehicle's surroundings with at least one visual camera having a field of view; and acquiring a thermal image of the vehicle's surroundings with at least one thermal camera having a field of view. Superimposing the thermal image on the visual image to produce a composite image. Detecting an object in the composite image. A vehicle assistance system adjusts at least one of a direction and a speed of travel of the vehicle in response to detecting the object.

[0004] In another aspect, a vehicle assistance system for a host vehicle traveling on a road includes a visual camera for acquiring a visual image of the host vehicle's surroundings in a field of view. A thermal camera acquires a thermal image of the host vehicle's surroundings in the field of view. A rain sensor detects precipitation in the field of view. A controller is connected to the visual camera, the thermal camera, and the rain sensor. The controller overlays the thermal image onto the visual image to form a composite image for detecting an object in the field of view. The controller adjusts at least one of a driving direction and a speed of the vehicle in response to detecting the object.

[0005] Other objects and advantages and a more complete understanding of the present invention will be obtained from the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a top view of a vehicle having an assist system according to an embodiment of the present invention.

[0007] Figure 2 yes Figure 1 Schematic diagram of the auxiliary system.

[0008] Figure 3A yes Figure 1 Schematic diagram of an example imaging element of an auxiliary system.

[0009] Figure 3B yes Figure 1 Schematic diagram of another example imaging element of an auxiliary system.

[0010] Figure 4 is a schematic diagram of a vehicle traveling on a road and detecting objects in front of the vehicle within a field of view.

[0011] Figure 5A is the visual image of the field of view.

[0012] Figure 5B is the thermal image of the field of view.

[0013] Figure 6 is a graph showing the correlation between visibility within the field of view and transparency of the thermal image.

[0014] Fig. 7A is a composite image of the thermal image having a first transparency and superimposed on the visual image.

[0015] Figure 7B is a composite image of the thermal image having a second transparency and superimposed on the visual image.

[0016] Figure 8 is a schematic diagram of a vehicle traveling on a road and detecting a traffic pattern indicator in front of the vehicle within a field of view.

[0017] Fig. 9 is a flow chart illustrating a method of assisting a vehicle. DETAILED DESCRIPTION

[0018] The present invention relates generally to vehicle systems and, more particularly, to a vehicle assistance system having an imaging element that takes into account changes in visibility around the vehicle. Figure 1 A vehicle 20 is shown having an exemplary assistance system 50 according to the present invention.

[0019] The vehicle 20 extends along a centerline 22 from a front end 24 to a rear end 26. The vehicle 20 includes a left side 27 and a right side 29 located on opposite sides of the centerline 22. A side view mirror 28 is connected to the left side 27. Another side view mirror 30 is connected to the right side 29.

[0020] The front end 24 of the vehicle 20 includes a front window or windshield 32 extending generally between a left side 27 and a right side 29. A rearview mirror 34 is secured to the windshield 32. The rear end 26 of the vehicle 20 includes a rear window 36 extending generally between the left side 27 and the right side 29. The vehicle 20 includes an interior 54. The exterior of the vehicle 20 is generally indicated at 56.

[0021] The vehicle 20 includes a pair of front steerable wheels 60 and a pair of rear wheels 62. The front wheels 60 are mechanically linked to a steering actuator or gear 68 (see Figure 2 ), the steering actuator or gear 68 is mechanically linked to the steering wheel 66. Alternatively, the front wheels 62 and the steering wheel 66 can be part of a steer-by-wire system (not shown). The rear wheels 62 can also be coupled to the steering wheel 66 via the same steering gear 68 or another steering gear (not shown).

[0022] In any case, rotation of the steering wheel 66 actuates the steering gear 68 to turn the wheels 60 relative to the centerline 22 in order to steer the vehicle 20. To this end, the steering wheel 66 has a neutral position in which the wheels 60 point in a direction parallel to the centerline 22 so that the vehicle moves in a straight line. Counterclockwise rotation of the steering wheel 66 causes the wheels 60 to be angled to the left relative to the centerline 22 (e.g., Figure 1 ), causing the vehicle 20 to turn left. Clockwise rotation of the steering wheel 66 causes the wheels 60 to be angled rightward relative to the centerline 22, causing the vehicle 20 to turn right.

[0023] The assist system 50 includes imaging elements 70a-70d disposed about the vehicle 20. In one example, the imaging element 70a is secured to the rearview mirror 34. The imaging element 70b is secured to the left side 27 on the sideview mirror 28. The imaging element 70c is secured to the right side 29 on the sideview mirror 30. The imaging element 70d is secured to the rear end 26 of the vehicle 20 along or adjacent to the centerline 22.

[0024] All imaging elements 70a-70d face outwardly away from the vehicle 20. Thus, imaging element 70a faces forward or toward the front. Imaging element 70d faces rearward or toward the rear. Imaging elements 70b, 70c face sideways or toward the sides. It should be understood that more or fewer imaging elements may be provided. In any case, all of imaging elements 70a-70d are electrically or wirelessly connected to a controller 74 in the vehicle 20.

[0025] Each imaging element 70a-70d has an associated field of view 72a-72d covering a portion of the vehicle exterior 56. In general, the fields of view 72a-72d substantially surround the entire vehicle 20 and may be discrete from one another (as shown) or overlap one another (not shown). The controller 74 continuously receives images captured by one or more of the imaging elements 70a-70d within the corresponding fields of view 72a-72d. The controller 74 includes an image processing module (not shown) that receives and analyzes data associated with the images from the imaging elements 70a-70d. The controller 74 may, for example, stitch the images together to form a 360° surround view of the vehicle exterior 56 (not shown). The images may be relied upon to identify objects around the vehicle 20. In some cases, less than all of the imaging elements 70a-70d are used to detect objects around the vehicle 20 and / or assist the operator.

[0026] refer to Figure 2 , the controller 74 is also electrically or wirelessly connected to various sensors and actuators in the vehicle 20 for monitoring and controlling several functions of the vehicle, namely vehicle speed and steering. To this end, the controller 74 is electrically or wirelessly connected to a vehicle speed sensor 100. The speed sensor 100 monitors the vehicle speed and generates an electrical signal 102 indicative of the vehicle speed, which is sent to the controller 74 at predetermined time intervals.

[0027] The controller 74 is also electrically or wirelessly connected to an actuator 110 associated with a vehicle brake 112 and a throttle actuator 116 associated with an accelerator pedal 118. The controller 74 can send a control signal 114 to the brake actuator 110 to reduce the speed of the vehicle 20. The controller 74 can send a control signal 120 to the throttle actuator 116 to increase the speed of the vehicle 20.

[0028] The wheel position sensor 130 monitors the rotation angle of the steering wheel 66 and generates an electrical signal 132 indicating the steering angle. The signal 132 is sent to the controller 74 at predetermined time intervals. The controller 74 can send a control signal 136 to the steering gear 68 in response to the wheel position signal 132, thereby controlling the rotation of the steering wheel 66. Actuation of the steering gear 68 also controls the steering angle of the front wheels 60 relative to the centerline 22 of the vehicle 20.

[0029] At least one light sensor 140 is electrically or wirelessly connected to the controller 74 for acquiring data related to the intensity of light around the vehicle exterior 56. One light sensor 140 is fixed to the rearview mirror 34 and has a detection range that substantially includes the field of view 72a of the imaging element 70a. The light sensor 140 fixed to the rear end 26 has a detection range that includes the field of view 72d of the imaging element 70d. The light sensors 140 are fixed to the left side 27 and the right side 29 of the vehicle 20 and have respective detection ranges that include the fields of view 72c-72d. The light sensors 140 detect the presence and intensity of light in the fields of view 72a-72d of the imaging elements 70a-70d. This would include, for example: light directly from the sun; sunlight reflected from the road, other vehicles, buildings, etc.; light from the headlights of oncoming vehicles. The light sensor 140 sends a signal 142 indicating the light intensity within the fields of view 72a-72d to the controller 74.

[0030] At least one proximity sensor 150 may be electrically or wirelessly connected to the controller 74 for acquiring data related to objects around the vehicle exterior 56. The at least one proximity sensor 150 may include, for example, a laser scanner, an ultrasonic sensor, a radar detector, and a LIDAR (light radar) detector for determining and monitoring the distance between the vehicle 20 and objects around the vehicle exterior 56 detected by the imaging elements 70a-70d. In one example, the proximity sensors 150 are disposed on the front end 24 and the rear end 26 of the vehicle 20. However, the proximity sensors 150 may be omitted entirely.

[0031] At least one rain sensor 170 can be electrically or wirelessly connected to the controller 74 for acquiring data related to precipitation around the vehicle exterior 56. As shown, the rain sensors 170 are disposed on the front end 24 and rear end 26 of the vehicle. The rain sensors 170 detect the presence of precipitation, such as rain, snow, hail, or fog, within the field of view 72a, 72d, and detect the intensity of the precipitation, such as mm / hr or in / hr, when precipitation is present. The rain sensor 170 sends a signal 172 indicating precipitation within the field of view 72a, 72d to the controller 74.

[0032] Based on this structure, the controller 74 is able to receive continuous feedback about the driving conditions of the vehicle (e.g., vehicle speed and steering angle, images around the vehicle exterior 56, weather conditions and lighting conditions around the vehicle, and the distance between the vehicle and objects identified in the image). In response to these inputs, the controller 74 can control vehicle operation in a manner that helps increase occupant safety. To this end, the controller 74 can assist or perform lane keeping assist emergency braking and backup assist in response to the images sent by the imaging elements 70a-70d.

[0033] The alarm 160 is electrically or wirelessly connected to the controller 74 for providing feedback to the operator of the vehicle 20 before and / or when the assistance system 50 performs autonomous operation. The alarm 160 provides visual, audible, or tactile feedback to the operator before and / or when the controller 74 sends the signal 142 to the operator.

[0034] Reference Figure 3A , an example imaging element 70a includes a housing 80 extending from a first end 82 to a second end 84 and defining an interior 86. The first end 82 is closed by an end wall 88. A visual camera 90 and a separate thermal camera 92 are disposed in the interior 86. Each camera 90, 92 has an associated lens 91, 93 disposed in the end wall 88 so that the cameras have substantially the same field of view 72a extending through the end wall in front of the vehicle 20. The visual camera 90 may be a single lens camera or a multi-focal camera.

[0035] Because the visual camera 90 relies on capturing light to generate an image, the visual camera is better suited than the thermal camera 92 for detecting inanimate objects, such as lane lines, parked vehicles, stop signs, debris in the road, etc. On the other hand, the thermal camera 92 relies on detecting heat to generate an image, and therefore, the thermal camera is better suited than the visual camera 90 for detecting living objects (such as pedestrians or animals), or objects that generate more heat relative to their surroundings (such as vehicle exhaust pipes).

[0036] exist Figure 3B In the alternative configuration shown, the visual camera 92 and the thermal camera 94 are integrated with each other, where both use the same lens 96 disposed in the end wall 88 to share the same field of view 72a. In any case, the imaging element 70a can have a single lens or a multiple lens configuration. It should be understood that the imaging elements 70b-70d can each have the same Figure 3A or Figure 3B The imaging element 70a has the same structure as in FIG.

[0037] In one example, the assistance system 50 detects objects within the field of view 72a and selects an appropriate response, if any, to the detection of those objects. To this end, the example road 200 is Figure 4 2 and has a vehicle travel direction shown by arrow T. Road 200 includes a series of lanes 202, 204 separated by a dashed dividing line 206. Additional lanes and dividing lines are contemplated but not shown. Road 200 is defined from surrounding off-road terrain 210 by a boundary line 212 on the left side (relative to the travel direction T) and a boundary line 214 on the right side.

[0038] refer to Figure 4As vehicle 20 travels along road 200, imaging element 70a detects objects in front of the vehicle within field of view 72a, such as other vehicles, pedestrians, animals, debris, etc. An example object is Figure 4 More specifically, the controller 74 continuously receives a signal representing a visual image from the visual camera 90 and a signal representing a thermal image from the thermal camera 92 .

[0039] Figure 5A 20. In the image 220, the lines 206, 212, 214 of the road 200 are visible. The indicator 222 approximates the location of what the controller 74 believes to be an object 216 (here, a pedestrian on the off-road terrain 210). The placement of the indicator 222 may be based on a neural network shape detection analysis performed by segmenting the visual image 220. The controller 74 may perform a pixel analysis within the indicator 222 and determine a first confidence score indicating the likelihood that the object 216 in the indicator is actually what the controller estimates it to be.

[0040] Figure 5B 2. An example of a thermal image 230 is shown in FIG. 2. In the image 230, the lines 206, 212, 214 of the road 200 are less visible and more difficult to detect than in the visual image 220. The indicator 232 approximates the location of what the controller 74 believes to be the object 216. The placement of the indicator 232 may be based on a fuzzy system shape detection analysis in a relative hot spot within the visual image 220. The controller 74 may perform a pixel analysis within the indicator 232 and determine a second confidence score indicating the likelihood that the object 216 in the indicator is actually what the controller estimates it to be. In the thermal image 230, the object 216 is more visible and easier to detect than in the visual image 220, and therefore, the second confidence score is greater than the first confidence score.

[0041] To utilize the detection capabilities of each camera 90, 92, the controller 74 superimposes or overlaps the visual image 220 and the thermal image 230 upon each other to form a composite image 240 ( Fig. 7A and Figure 7B ). This allows the controller 74 to utilize the capabilities of both cameras 90 , 92 to increase the reliability of the assistance system 50 in detecting objects around the vehicle 20 .

[0042] However, weather conditions and lighting conditions surrounding the vehicle may affect the ability of the visual camera 90 to detect objects within the field of view 72a. Therefore, the controller 74 relies on one or more of the sensors 140, 170 to detect and monitor weather conditions and lighting conditions surrounding the vehicle 20. The controller 74 assumes the absence of any adverse weather conditions or adverse lighting conditions to equate to high visibility of the visual camera 90 within the field of view 72a. On the other hand, the controller 74 assumes an indication of adverse weather conditions and / or adverse lighting conditions to equate to low visibility of the visual camera 90 within the field of view 72a. In response to such an indication (or lack of an indication), the controller 74 adjusts the transparency of the thermal image 230 before superimposing the thermal image on the visual image 220.

[0043] In one example, if the detected light intensity is within a predetermined range consistent with what is considered normal lighting conditions, the transparency of the thermal image 230 is increased. If the detected light intensity is below the predetermined range (considered too dark) or above the predetermined range (considered too bright), the transparency of the thermal image 230 is decreased. The more extreme the light intensity (or lack thereof), the lower the transparency of the thermal image 230.

[0044] Along the same lines, if no precipitation is detected or the detected precipitation is at or below a predetermined amount, the transparency of the thermal image 230 increases. If precipitation is detected or the detected precipitation is above a predetermined amount, the transparency of the thermal image 230 decreases. The greater the precipitation sensed, the lower the transparency of the thermal image 230.

[0045] The controller 74 receives the signals 142, 172 from the sensors 140, 170 and determines an overall visibility score, e.g., 0-100%, for the visual image 220 based on an algorithm. In one example, the controller 74 assigns a first visibility score and a second visibility score to each sensed weather condition and lighting condition around the vehicle 20 based on a lookup table listing possible weather conditions and lighting conditions and their respective corresponding visibility scores. The first visibility score and the second visibility score may then be weighted and averaged to determine an overall visibility score. Certain weather conditions or lighting conditions may be weighted more heavily than other conditions based on their known impact on the visibility of the visual image.

[0046] Once the total visibility score is established, the controller 74 adjusts the transparency of the thermal image 230. The greater the total visibility score, the greater the transparency of the thermal image 230. The lower the total visibility score, the lower the transparency of the thermal image 230. In other words, the degree of visibility in the visual image 220 is directly related to the degree to which the thermal image 230 becomes transparent. This relationship is Figure 6 Shown in.

[0047] Fig. 7A and Figure 7B Composite images 240 are shown with varying degrees of transparency of thermal image 230. Fig. 7A , the total visibility score of visual image 220 is 75%, and therefore thermal image 230 has a transparency of 25%. This corresponds to a relatively high visibility in visual image 220, and therefore controller 74 relies relatively less on thermal image 230 to detect object 216 and lines 206, 212, 214. Figure 7B , the total visibility score of visual image 220 is 55%, and thus thermal image 230 has a transparency of 45%. This corresponds to a relatively low visibility in visual image 220, and thus controller 74 relies more on thermal image 230 to detect object 216 and lines 206, 212, 214.

[0048] In each case, the object 216 and the lines 206, 212, 214 are more easily detected in the synthetic image 240C than relying only on the visual image 220 or only on the thermal image 230. In the synthetic image 240, the lane lines 206, 214 in each image 220, 230 overlap each other. The indicators 222, 232 overlap each other. The controller 74 can perform pixel analysis within the overlapping indicators 222, 232 and determine a third confidence score indicating the likelihood that the object 216 in the indicator is actually what the controller estimates it to be. The third confidence score is greater than the first confidence score and the second confidence score. It should be understood that the controller 74 can adjust the first confidence score, the second confidence score, and / or the third confidence score based on the sensed weather conditions and lighting conditions around the vehicle 20, for example, the confidence score decreases as weather conditions deteriorate and / or when the lighting is poor.

[0049] Once object 216 is detected in composite image 240, controller 74 relies on proximity sensor 130 to monitor the distance between vehicle 20 and the object. In response to detecting object 216, controller 74 is configured to adjust the speed of vehicle 20 as needed. In one example, if controller 74 determines that object 216 is within a predetermined distance d1 ( Figure 4 ), the controller initiates emergency braking by sending a control signal 114 to the brake actuator 110 to reduce the speed of the vehicle 20. The combination of the speed of the vehicle 20 and the distance to the object may cause the controller 74 to bring the vehicle to a complete stop. This may occur, for example, when the object 216 is located in front of the vehicle 20 and within the lane 204 (not shown).

[0050] Before initiating emergency braking, the controller 74 may consider the first confidence score and the second confidence score and / or the third confidence score. For example, emergency braking does not occur until / unless the considered confidence score exceeds a predetermined value.

[0051] It should be understood that the distance d1 may be detected and monitored in each of the images 220, 230, 240. Based on the vehicle speed signal 102, the controller 74 may determine a time of collision with the object 216. The time of collision calculated from the visual image 220, the thermal image 230, and / or the synthetic image 240 may be considered when the controller 74 decides whether to initiate emergency braking. A confidence value may be assigned to the time of collision determination, and the confidence value may be adjusted based on sensed weather conditions and / or lighting conditions.

[0052] Although object 216 is shown as a pedestrian, the object may equally be another vehicle within lane 204 in front of vehicle 20 (when the vehicle is traveling in direction T) or behind the vehicle (when the vehicle is backing up). In the latter case, controller 74 relies on imaging element 70d and sensors 140, 170 on rear end 26 of vehicle 20 to identify objects in field of view 72d. In either case, the other vehicle may be moving or stationary. In other words, synthetic image 240 may be generated while vehicle 20 is traveling in the forward or rearward direction to detect moving or stationary objects in either field of view 72a, 72d. Controller 74 may use either confidence score and / or collision time determination to determine whether to initiate emergency braking in these cases.

[0053] At the same time, the proximity sensor 150 is used in conjunction with the imaging element 70a to allow the controller 74 to provide lane keeping assistance. When the vehicle 20 is traveling in the direction T, the controller 74 relies on the synthetic image 240 and the proximity sensor 150 to monitor the distance between the vehicle 20 and each line 206, 214. In response to detecting the object 216, the controller 74 is configured to adjust the driving direction T of the vehicle 20 when necessary.

[0054] For example, wheel position sensor 130 continuously supplies signal 132 to controller 74. As a result, controller 74 can analyze synthetic image 240 and signal 152 from proximity sensor 150 and provide autonomous lane keeping assistance. In particular, controller 74 can actuate steering gear 68 to prevent vehicle 20 from unintentionally drifting into lane 202 on demarcation line 206. This may occur, for example, if the operator of vehicle 20 is distracted, fatigued, etc.

[0055] For example, if the proximity sensor 150 detects that the vehicle 20 is within a predetermined distance from the dividing line 206, the controller 74 actuates the steering gear 68 to rotate the steering wheel 66 clockwise from the neutral position. This pivots the wheels 60 and causes the vehicle 20 to move laterally toward the boundary line 214. Once the proximity sensor 150 indicates that the vehicle 20 is separated from the two lines 206, 214 by a desired distance, the controller 74 returns the steering wheel 66 to the neutral position so that the vehicle 20 travels in a straight line in the lane 204 in the direction T.

[0056] Similarly, if the proximity sensor 150 detects that the vehicle 20 is within a predetermined distance from the boundary line 214, the controller 74 actuates the steering gear 68 to rotate the steering wheel 68 counterclockwise from the neutral position. This pivots the wheels 60 and causes the vehicle 20 to move laterally toward the dividing line 206. Once the proximity sensor 150 indicates that the vehicle 20 is spaced a desired distance from the two lines 206, 214, the controller 74 returns the steering wheel 66 to the neutral position so that the vehicle 20 travels in a straight line in the lane 204 in the direction T. In both cases, the controller 74 sends a signal 162 to the alarm 160 to provide feedback to the operator before and / or when autonomous steering corrections are made to keep the vehicle 20 between the lines 206, 214.

[0057] It should be understood that the controller 74 can also be configured so that once an object is detected, the controller can track and / or classify the object to better determine when the driving direction T and / or speed needs to be adjusted. To this end, the controller 74 can use a tracking algorithm that continuously monitors the distance d1 between the detected object and the vehicle 20 and specific features of the detected object, such as size, shape, heat intensity, speed, etc. The controller 74 can then track changes in the distance d1 and / or specific features over time in order to classify the detected object as, for example, a person, an animal, a vehicle, etc. This allows the controller 74 to specifically customize the adjustment of the vehicle 20 based on the classification of the object. It also allows the controller 74 to determine the nearest in-path vehicle at this time and take appropriate countermeasures if necessary.

[0058] exist Figure 8 In another example shown, the object detected within the field of view 72a is a traffic pattern indicator 266, such as a traffic light. Other example traffic pattern indicators 266 may include, but are not limited to, a stop sign, a yield sign, a construction zone indicator, a school zone indicator, a siren, etc. In any case, the vehicle 20 travels along the road 200 in the direction T. Another road 262 extends transversely to the road 200 and intersects the road 200 at the intersection 260. As shown, a traffic light 266 is suspended above the roads 200, 262 at the intersection 260.

[0059] As the vehicle 20 approaches the intersection 260, the controller 74 continuously receives images from the two cameras 90, 92 in the imaging element 70a and signals from the sensors 140, 170. The controller 74 relies on the composite image 240 to identify when the traffic light 266 is within the field of view 72a and what color the light is based on the sensed light and / or heat emitted from the currently activated light. The controller 74 can control the vehicle system as appropriate in response to detecting the traffic light 266. For example, the controller can actuate the brake actuator 110 based on the detection of a red traffic light 266, the distance between the vehicle 20 and the traffic light, and the vehicle speed. Therefore, if a red light is detected, the controller 74 can decelerate the vehicle 20 downward until and including a complete stop before the intersection 260. The controller 74 can take similar steps in response to detected stop signs, school zones, construction zones, etc.

[0060] Fig. 9 An example method 300 of operating the assist system 50 is shown. At step 302, the controller 74 acquires the visual image 220 from the visual camera 90. At step 304, the controller 74 acquires the thermal image 230 from the thermal camera 92. At step 310, the controller 74 detects objects in the visual image 220 using, for example, a neural network shape detection analysis. At step 312, the controller 74 detects objects in the thermal image 230 using, for example, a fuzzy system shape detection analysis. At step 320, the controller 74 overlays the thermal image 230 onto the visual image 220 to form a composite image 240. Before overlaying the thermal image onto the visual image 220, the transparency of the thermal image 230 is adjusted based on the sensed weather conditions and environmental conditions.

[0061] At step 330, the controller 74 determines the distance d1 between the vehicle 20 and the detected object. At step 340, the controller 74 uses a tracking algorithm to track the detected object. At step 350, the controller 74 classifies the detected object. The information obtained and determined by the controller 74 during the outlined steps can be used to adjust the driving direction T of the vehicle 20, for example, to perform lane keeping assistance or avoid detected objects, and / or adjust the speed of the vehicle. It should be understood that the same method 300 can be performed on any number of images 220, 230 acquired by the controller 74 from the corresponding cameras 90, 92. In other words, the image acquisition, manipulation and analysis described herein can be performed on a pair of images 220, 230 or on multiple pairs of images continuously to achieve real-time monitoring and feedback the surrounding environment of the vehicle 20 to the controller 74.

[0062] What has been described above is an example of the present invention. Of course, for the purpose of describing the present invention, it is not possible to describe every conceivable combination of components or methods, but those of ordinary skill in the art will recognize that many other combinations and permutations of the present invention are possible. Therefore, the present invention is intended to cover all such changes, modifications and variations that fall within the spirit and scope of the appended claims.

Claims

1. A method of assisting the operation of a vehicle travelling on a road, include: acquiring a visual image of the vehicle's surroundings with at least one visual camera having a field of view; acquiring a thermal image of a surrounding of the vehicle with at least one thermal camera having the field of view; superimposing the thermal image on the visual image to produce a composite image; detecting an object in the composite image; as well as In response to detecting the object, adjusting at least one of a direction and a speed of the vehicle, The method further comprises: sensing weather conditions and lighting conditions in the field of view; assigning a first visibility score to the visual image based on the sensed weather condition; assigning a second visibility score to the visual image based on the sensed lighting conditions; combining the first visibility score and the second visibility score into a total visibility score; and Prior to superimposing the thermal image into the visual image, a transparency of the thermal image is adjusted based on the total visibility score. 2 . The method of claim 1 , wherein combining the first visibility score and the second visibility score comprises calculating a weighted average of the first visibility score and the second visibility score to determine the total visibility score.

3. The method according to claim 1, further comprising: include: Detecting lane lines of the road using the synthetic image; as well as Lane keeping assist is autonomously performed in response to detecting the lane marking.

4. The method according to claim 1, further comprising: include: detecting objects on the road using the composite image; as well as The vehicle is autonomously braked in response to detecting the object.

5. The method according to claim 1, further comprising: include: detecting traffic pattern indicators along the roadway using the composite image; as well as The vehicle is autonomously braked in response to detecting the traffic pattern indicator.

6. A vehicle assistance system for assisting a host vehicle traveling on a road using the method according to any one of claims 1 to 5, include: a visual camera for acquiring a visual image of the surroundings of the host vehicle in a field of view; a thermal camera for acquiring a thermal image of the surroundings of the host vehicle in the field of view; a controller connected to the visual camera and the thermal camera and configured to superimpose the thermal image onto the visual image to form a composite image for detecting an object within the field of view and adjusting at least one of a direction and a speed of travel of the vehicle in response to detecting the object; as well as A rain sensor coupled to the controller for detecting precipitation within the field of view, the controller being configured to adjust a transparency of the thermal image in response to an amount of precipitation detected by the rain sensor within the field of view before superimposing the thermal image on the visual image.

7. The vehicle assistance system of claim 6 further comprising a light sensor connected to the controller, the light sensor being used to detect light intensity in the field of view, the controller being configured to adjust the transparency of the thermal image in response to the light intensity detected by the light sensor within the field of view before superimposing the thermal image on the visual image. 8 . The vehicle assistance system of claim 6 , wherein the controller is configured to perform at least one of lane keeping assistance and braking in response to detecting the object.

Images