Obstacle detection for wiper wear monitoring

Abstract

The present disclosure provides "occlusion detection for wiper wear monitoring". A vehicle includes a camera configured to capture an image through a window of the vehicle; a vehicle window cleaning system operable between a cleaning operation and a cleaning operation and including: a wiper configured to move along a vehicle window in the cleaning operation and the cleaning operation; and an interface for manually activating the cleaning operation. A control circuit configured to: detect an obstruction on the vehicle window based on the image; classifying the shelter as water or debris; detecting a manual operation on the wiper blade; determining a wear condition during manual operation of the wiper blade and based on classifying the shield as debris; and transmitting a signal indicative of the wear condition.

Description

Technical Field

[0001] The present disclosure generally relates to image processing in a vehicle environment, and more particularly to occlusion detection for wiper wear monitoring. Background Art

[0002] Conventional image processing techniques for detecting moisture conditions on or around a vehicle may be limited. Summary of the Invention

[0003] According to a first aspect of the present disclosure, a vehicle includes: a camera configured to capture an image through a window of the vehicle; a window cleaning system operable between a cleaning operation and a washing operation and including: a wiper configured to move along the window in the cleaning operation and the washing operation; and an interface for manually activating the cleaning operation. A control circuit configured to: detect an occlusion on the window based on the image; classify the occlusion as water or debris; detect a manual operation of the wiper; determine a wear condition during the manual operation of the wiper and based on classifying the occlusion as debris; and transmit a signal indicating the wear condition.

[0004] Embodiments of the first aspect of the present disclosure may include any one or a combination of the following features:

[0005] - The window cleaning system includes a sprayer configured to apply a cleaning fluid to the window in the washing operation;

[0006] - The control circuit includes a memory configured to store wear information including the usage time of the wiper;

[0007] - The usage time includes the duration of the wear condition;

[0008] - The window cleaning system is operable in an automatic mode in which the cleaning operation and the washing operation are automatically activated in response to detecting the occlusion;

[0009] - The window cleaning system includes a pump for pressurizing the cleaning liquid to apply it to the window in the washing operation;

[0010] - Determine the cleaning operation and the washing operation in the automatic mode based on classifying the occlusion as water or debris;

[0011] - The window cleaning system is configured to operate the wiper in response to classifying the occlusion as water and operate the sprayer in response to classifying the occlusion as debris;

[0012] - The control circuit is configured to control the window cleaning system to adjust from the cleaning operation to the cleaning operation in response to the wear condition;

[0013] - The control circuit is configured to classify the obscuration based on an optical distortion level, wherein classifying the obscuration as water or debris is based on the optical distortion level;

[0014] - The control circuit is configured to capture a subsequent image of the window after the manual operation; and classify the wear condition based on the subsequent image with a modifier;

[0015] - The modifier is a calculated multiplier for the wear condition, and wherein the control circuit is configured to select the multiplier from a plurality of modifiers corresponding to environmental conditions;

[0016] - The control circuit is configured to transmit a signal to apply a cleaning fluid to the window in response to the classification of the wear condition with the modifier; and

[0017] - A notification device configured to indicate the wear condition in response to the signal.

[0018] According to a second aspect of the present disclosure, a vehicle includes: a camera configured to capture an image through a window of the vehicle; a window cleaning system operable between a cleaning operation and a cleaning operation and including: a sprayer configured to apply a cleaning fluid to the window during the cleaning operation; a wiper configured to move along the window during the cleaning operation and the cleaning operation; and an interface for manually operating the wiper. A control circuit configured to: detect an obscuration on the window based on the image; classify the obscuration as water or debris; detect a manual operation of the wiper; determine a wear condition during the manual operation of the wiper and based on classifying the obscuration as debris; and transmit a signal indicating the wear condition.

[0019] Embodiments of the second aspect of the present disclosure may include any one or a combination of the following features:

[0020] - The window cleaning system is operable in an automatic mode in which the cleaning operation and the cleaning operation are automatically activated in response to detecting the obscuration;

[0021] - The window cleaning system includes a pump for pressurizing the cleaning liquid for application to the window during the cleaning operation;

[0022] - Determine a selection between the cleaning operation and the clearing operation in the automatic mode based on classifying the obscuration as water or debris; and

[0023] - The window clearing system is configured to operate the wiper in response to classifying the obscuration as water and operate the sprayer in response to classifying the obscuration as debris.

[0024] According to a third aspect of the present disclosure, a vehicle includes: a camera configured to capture an image through a window of the vehicle; a window clearing system operable between a clearing operation and a cleaning operation and including: a sprayer configured to apply a cleaning fluid to the window in the cleaning operation; a wiper configured to move along the window in the clearing operation and the cleaning operation; and an interface for manually operating the wiper; a notification device configured to indicate a wear condition of the wiper in response to a signal. A control circuit configured to: detect an obscuration on the window based on the image; classify the obscuration as water or debris; detect a manual operation of the wiper; determine the wear condition during the manual operation of the wiper and based on classifying the obscuration as debris; and transmit the signal indicating the wear condition.

[0025] Those skilled in the art will further understand and appreciate these and other features, advantages, and objectives of the present disclosure by referring to the following specification, claims, and drawings. Description of the Drawings

[0026] In the drawings:

[0027] Figure 1 is a perspective view of a vehicle incorporating a moisture detection system according to one aspect of the present disclosure;

[0028] Figure 2 is a functional block diagram of a moisture detection system of a vehicle according to one aspect of the present disclosure;

[0029] Figure 3A is an image of dust on a window to be cleaned by a window clearing system of the present disclosure;

[0030] Figure 3B is an image of biological residue of an insect on a window to be cleaned by a window clearing system of the present disclosure;

[0031] Figure 4 is a flowchart of an automatic mode for a window clearing system of the present disclosure to clean and / or clear a window of a vehicle;

[0032] Figure 5is an exemplary cross-sectional view of a vehicle incorporating, adjacent to a windshield within the interior of the vehicle, an imaging device for detecting a moisture condition in an external area of the vehicle;

[0033] Figure 6A is an exemplary image captured by the imaging device positioned away from the windshield within the interior of the vehicle such that water droplets on the windshield can be detected;

[0034] Figure 6B is an exemplary image captured by an imaging device positioned within the interior of the vehicle and near the windshield, showing detected water streaks on the windshield;

[0035] Figure 7 is a representation of a fully convolutional data description (FCDD) network that processes an image captured by an imaging device positioned near the windshield and generates image data showing occlusions in the image;

[0036] Figures 8A to 8C is the captured image and the image data showing occlusions on the vehicle's windshield after being processed by the FCDD network;

[0037] Figures 9A to 9B is an exemplary image captured by the vehicle's imaging device showing a splash event along the vehicle's road, where Figure 9B shows a splash zone covering the splash event;

[0038] Figure 10A is an exemplary depiction of a radar scan of the vehicle's external environment capturing a splash condition;

[0039] Figure 10B is an exemplary depiction of a radar scan of the vehicle's external environment capturing a splash condition;

[0040] Figure 11 is a block diagram showing an exemplary overtaking condition detection based on splash monitoring in the vehicle's environment; and

[0041] Figure 12 is an exemplary process performed by a moisture detection system in accordance with one aspect of the present disclosure. Detailed Description

[0042] Reference will now be made in detail to the preferred embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or like parts. In the drawings, the structural elements depicted are not drawn to scale, and for purposes of emphasis and understanding, certain components are enlarged relative to other components.

[0043] As needed, detailed embodiments of the present disclosure are disclosed herein; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure, which may be implemented in various and alternative forms. The accompanying drawings are not necessarily to scale; some of the schematic diagrams may be enlarged or minimized to show a functional overview. Accordingly, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to employ the present disclosure in different ways.

[0044] For purposes of description herein, the terms "up", "down", "right", "left", "rear", "front", "vertical", "horizontal", and derivatives thereof shall relate to the concepts as oriented in Figure 1 the figures. However, it should be understood that unless explicitly specified to the contrary, the said concepts may assume various alternative orientations. It should also be understood that the specific apparatus and processes shown in the drawings and described in the following specification are merely exemplary embodiments of the inventive concepts defined in the appended claims. Accordingly, unless the claims otherwise explicitly state, the specific dimensions and other physical characteristics related to the embodiments disclosed herein should not be considered limiting.

[0045] The illustrated embodiments mainly exist in the combination of method steps and device components related to the detection of obstacles for wiper wear monitoring. Accordingly, the device components and method steps have been represented by conventional symbols in the drawings where appropriate, showing only those specific details relevant to understanding the embodiments of the present disclosure, so as not to obscure the present disclosure with details that are obvious to those of ordinary skill in the art who benefit from the description herein. In addition, the same reference numerals in the specification and drawings denote the same elements.

[0046] Generally referring to Figures 1 to 12, the moisture detection system 10 uses image processing to detect the condition of an area 12 outside the vehicle 14, such as water on the outer surface 16 of the windshield 18 of the vehicle 14 or water in front of or around the vehicle 14. Generally, the systems and methods of the present disclosure can provide enhanced intensity classification for splashes in the area 12 outside the vehicle 14 by combining depth detection or density detection using radio detection and ranging (radar). Additionally, the systems and methods of the present disclosure can provide enhanced space utilization within the vehicle 14 by allowing the imaging device 20 of the moisture detection system 10 to be positioned near or away from the windshield 18 while still detecting and classifying obstructions on the windshield 18. Further, the moisture detection system 10 can provide enhanced visibility by optimizing the cleaning and / or manipulation of the windshield 18 or suggesting maneuvering the vehicle 14 to a target location to enhance visibility and / or control. The moisture detection system 10 can also provide a more accurate detection of weather conditions, such as rain, humidity, fog, or other moisture conditions in the area 12 outside the vehicle 14, thereby allowing enhanced responsiveness to other vehicle systems. Additionally, the systems and methods of the present disclosure can provide increased lifespan of parts (e.g., window cleaning parts), efficient use of window cleaning fluid, and overall enhanced automatic window cleaning control.

[0047] Now referring to Figure 1 , the moisture detection system 10 for the vehicle 14 can incorporate an imaging device 20 positioned within the passenger compartment 22 of the vehicle 14. For example, the imaging device 20 can be positioned adjacent to the windshield 18 of the vehicle 14 within the passenger compartment of the vehicle 14 and oriented in a vehicle forward orientation. At least one windshield wiper 26 is positioned on the outer surface 16 of the windshield 18 for clearing water, debris, or other substances from the windshield 18. As will be further described with respect to Figure 2 , a window cleaning system can be employed to clean the windshield 18 or any of the multiple windows of the vehicle 14 where moisture or other substances can accumulate on the window from the outside area 12 of the vehicle 14. The window cleaning system can include a nozzle 28 positioned adjacent to the windshield wiper 26 between the hood 30 of the vehicle 14 and the windshield 18 of the vehicle 14 and configured to spray a cleaning fluid onto the windshield 18. The cleaning fluid can include a cleaning agent, such as methanol, ethylene glycol, or other fluids for cleaning substances on the outer surface 16 of the windshield 18 when used in conjunction with the windshield wiper 26. It is contemplated that the cleaning fluid can contain water, which in some examples can be heated to assist in the defrosting operation of the windshield 18.

[0048] Still referring to Figure 1, multiple distance sensors 34, 36, 38 are incorporated in the vehicle 14 to detect other objects in the external area 12 of the vehicle 14, such as other vehicles 14. The detection means may include ultrasonic and / or infrared detectors and a camera 38, such as an imaging device, which are configured to detect the distance from the vehicle 14 to surrounding objects in the external area 12 of the vehicle 14. Thus, the images from the imaging device 20 can be used to detect the distance from the vehicle 14 to the objects within the field of view of the imaging device.

[0049] As will be further described with respect to Figure 2 The distance sensors 34, 36, 38 can include any one of a radio detection and ranging sensor (radar 34), a light detection and ranging sensor (lidar 36), and a camera 38. In some examples, ultra-wideband (UWB) sensors are employed to detect objects in the external area 12. Generally, the distance sensors 34, 36, 38 can be configured to detect cross-traffic events and / or detect objects in the blind spot of the vehicle 14 to assist the user of the vehicle 14 in maneuvering the vehicle 14. In this example, the data collected by the detection sensors is used by the moisture detection system 10 to allow the moisture detection system 10 to detect the distance from the vehicle 14 to the splash event 118 (e.g., the following distance 120) and classify the importance, relevance, or priority of the splash event 118 based on the distance.

[0050] As will be further described herein, the distance sensors 34, 46, 38 can coordinate detection techniques by using one or more of the cameras 38 to detect in front of the splash event 118 and one or more of the radars 34 to detect the depth D and / or density of the splash event 118. For example, when there is significant visual occlusion in the captured image, image-based detection may be limited. However, one or more radars 34 can be used to transmit / receive radio waves or microwaves (e.g., via a radar transmitter and a radar receiver), which are reflected from the water droplets in the splash event 118 at a more precise level than the visible light waves received by the camera 38. For example, using the information from the radar 34, a control circuit in communication with the radar 34 can determine the depth D, density, intensity, span, or other properties of the splash event 118 based on the information. Generally, the radar 34 utilizes Doppler weather detection techniques to generate a map or distribution of the water in the splash event 118. For example, the radar 34 can transmit microwave or radio wave energy and measure the reflected waves from the splash event 118. Such measurements can include signals having a different frequency from the signal transmitted by the radar 34, resulting in a frequency shift. The frequency shift can be directly related to the movement (speed) of the raindrops or other droplets 108 in the splash event 118 or the precipitation. Thus, the stronger the rainfall or splash, the more water droplets 108 there are, and the stronger the echo signal detected by the radar 34.

[0051] Still referring to Figure 1 , vehicle 14 includes a plurality of wheels 40, each wheel having a tire 42 that interacts with a travel surface 44 of the vehicle 14. The friction between each tire 42 and the travel surface 44 can be affected by the moisture condition between the tire 42 and the travel surface 44. Accordingly, a moisture detection system 10 can be employed to control the drive of the wheels 40 to enhance the maneuverability of the vehicle 14 along the travel surface 44.

[0052] Vehicle 14 includes at least one lighting assembly 46, such as a headlight assembly having headlights 48 configured to illuminate an exterior area 12 of the vehicle 14. For example, the lighting assembly can be configured to illuminate the exterior area 12 of the vehicle 14 at a plurality of lighting levels (e.g., high beam, low beam, etc.). The moisture detection system 10 can enhance the control of the lighting assembly and its power level and / or lighting level. For example, the moisture condition in the exterior area 12 detected by the moisture detection system 10 can cause the moisture detection system 10 to control the power level of the lighting assembly due to reduced visibility from the moisture condition.

[0053] Now referring to Figure 2 , the moisture detection system 10 includes an imaging system 50, a distance detection system 52, and a response control system 54 in communication with the imaging system 50 and the distance detection system 52. Generally, data from the imaging system 50 and / or the distance detection system 52 is transmitted to the response control system 54, and the response control system 54 can control one or more visibility features and / or motion control features of the vehicle 14. For example, the response control system 54 includes one or more controllers 56, 72 having at least one processor and a memory in communication with the processor. The memory can store instructions that, when executed by the processor of the response control system 54, cause the response control system 54 to perform various tasks related to enhancing the visibility of the exterior area 12 of the vehicle 14 and / or the motion control of the vehicle 14. Generally, one or more of the controllers 56, 72 and / or other electrical components that provide decision-making via software can be referred to as control circuitry.

[0054] For example, the response controller 56 of the response control system 54 can include a motion control unit 58 and a visibility control unit 60. The visibility control unit 60 can be configured to control a window cleaning system, a lighting control system 62, and / or any other system that affects visibility through one or more windows of the vehicle 14. The motion control unit 58 can control communication with one or more vehicle systems, such as the powertrain 64 of the vehicle 14, the braking system 66 of the vehicle 14, or any other motion control system for the vehicle 14. For example, the motion control unit 58 can include a speed sensor in the powertrain 64 that detects rotation of a gear assembly in the powertrain 64. However, other speed sensors can be used to detect or infer the speed of the vehicle 14 (RF waves, inductive sensing, capacitive sensing, etc.). Additionally, the response control system 54 can include a display 68 (such as a human machine interface (HMI 70) in the cabin of the vehicle 14) and / or communicate therewith. The display 68 is configured to present messages to the user and / or allow the user to control the window cleaning system, the lighting control system 62, or any other aspect related to the visibility and / or motion of the vehicle 14. Generally, the response control system 54 can be configured to actively control the visibility and / or motion characteristics of the vehicle 14 or passively present messages at the display 68 to indicate visibility and / or motion target operations for the user to initiate.

[0055] Continuing to refer Figure 2 , the window cleaning system can include a window cleaning controller 72 that controls the operation of a pump 74 that pressurizes a cleaning fluid to eject the cleaning fluid through a nozzle 28 onto the windshield 18, as described previously. A valve 76 can be fluidly interposed between the pump 74 and the nozzle 28 to selectively allow the cleaning fluid to enter the nozzle 28.

[0056] The window cleaning controller 72 can also communicate with the motor 78 that drives the windshield wiper 26. For example, the motor 78 can be configured to rotate the windshield wiper 26 above the windshield 18 in response to a signal from the window cleaning controller 72. In some examples, the speed of the vehicle 14, as detected by a speed sensor, can be compared with the detected splash event 118, and in response to that detection, the motor 78 can be energized to operate at a specific number of revolutions per minute (RPM). At least one switch 80 communicates with the window cleaning controller 72 and / or communicates directly with the motor 78, the pump 74, and / or the valve 76 to control the dispensing of the cleaning fluid and / or the driving of the windshield wiper 26 via a manual interaction. For example, at least one switch 80 can include a first mechanism 82 that causes the cleaning fluid to be dispensed onto the windshield 18 and a second mechanism 84 that controls the operation of the windshield wiper 26. For example, the first mechanism 82 can be a knob that, when pulled or pushed, causes the cleaning fluid to be dispensed, and the second mechanism 84 can be a button or knob that moves the windshield wiper 26 above the windshield 18. It is contemplated that in some examples, the window cleaning controller 72 can be omitted, and the response control system 54 can directly control the window cleaning operation. In such an example, at least one of the switches 80 intervenes between the valve 76 and the nozzle 28 (e.g., the first mechanism 82), and at least one of the switches 80 intervenes between the motor 78 and the windshield wiper 26 (e.g., the second mechanism 84). In either example, the user can manually control the dispensing of the cleaning fluid and / or the operation of the windshield wiper 26, and such operations can also or alternatively be automatically controlled by the response control system 54. In another example, an instruction to initiate a wiping of the windshield 18 and / or a cleaning of the windshield 18 is represented at the HMI 70, and automatic control of the window cleaning system is omitted.

[0057] The visibility control unit 60 can also or alternatively be configured to control the lighting assembly of the vehicle 14 based on the moisture condition detected by the imaging system 50, as described above. For example, in the case where it is raining in the exterior area 12 of the vehicle 14, the headlights 48 can be automatically activated, or the response control system 54 can transmit an instruction to present a message at the display 68 to cause the user to activate the headlights 48 in response to the detected moisture condition in the exterior area 12 of the vehicle 14. Additionally, the brightness level (e.g., binary high beam / low beam or fine brightness control) can be controlled by the response control system 54 either actively or passively (e.g., presenting a message at the display 68).

[0058] Typically, the motion control unit 58 can control various systems related to the motion control of the vehicle 14, such as the drive of the wheels 40 (e.g., torque value), the brakes of the braking system 66 (e.g., traction control, anti-lock brakes (ABS)), the steering of the vehicle 14 (e.g., maneuvering along the driving surface 44), or any other motion control unit 58 for the vehicle 14. Similar to the operation of the visibility control unit 60, as a supplement or alternative to the automatic control of the maneuverability of the vehicle 14, the characteristics of the vehicle 14 related to motion control can be presented at the display 68. For example, in at least a partial semi-autonomous control mode of the vehicle 14, the visibility control unit 60 can transmit instructions to the user at the display 68 to reduce speed, maneuver the vehicle 14 left or right, etc., and in an alternative, can control speed, left or right maneuverability, etc. in response to detecting a moisture condition.

[0059] Continuing to refer Figure 2 , the various responses transmitted by the response control system 54 can be based on the output from one or both of the imaging system 50 and the distance detection system 52. Specifically referring to the imaging system 50, the imaging device 20 captures one or more images (captured images 86) of the external area 12 of the vehicle 14, and then the one or more images are processed by the image processor 88 to detect moisture conditions on or around the vehicle 14 and generate one or more output images 90. The image processor 88 communicates with a memory 91 storing instructions that, when executed, cause the image processor 88 to detect water patterns, water droplets 108, splash events 118, splash sources 116, or any other optical distortions related to moisture condition detection and / or occlusion detection. A fully convolutional data description network (FCDDNN 92) is included within or communicates with the memory 91, and the fully convolutional data description network can be a neural network that segments image data and detects optical distortions in the captured images 86. The FCDDNN 92 can be trained by a training module 94 of the imaging system 50, which provides sample images and / or historical image data presenting optical distortions (e.g., optical distortions caused by moisture in the image). The FCDDNN 92 is used to detect the portions of the image that are distorted due to the moisture condition and will be Figure 7 described in further detail.

[0060] In addition to detecting the moisture condition, the image processor 88 is also configured to detect any other occlusions within the field of view of the imaging device 20. For example, the occlusion can be environmental debris such as excrement, leaves, branches, non-water materials, or any other substance that may adhere to or fall on the outer surface 16 of the windshield 18. Thus, the imaging system 50 can distinguish debris from water. For example, the FCDDNN 92 can be trained to score occlusions based on a certain level of opacity, light transmittance, light distortion, etc. For example, debris may be associated with opacity, while water and / or other moisture conditions may be associated with the bending of light through the occlusion.

[0061] Generally, the response control system 54 can use the distance information from the distance detection system 52 and the moisture classification and detection from the image processing system to initiate a vehicle 14 response. For example, if moisture above a specific threshold (e.g., a low visibility threshold) is detected on the windshield 18, the imaging system 50 can transmit an output indicating the environmental condition of the external area to the response control system 54. For example, the imaging system 50 can determine that it is raining, snowing, or otherwise precipitating in the external area 12 and transmit an indication to activate the windshield wiper 26 to the response control system 54. The image processing system can further distinguish various forms of precipitation and / or occlusions on the windshield 18 to allow for personalized responses to be initiated by the control system based on the environmental condition. For example, and as will be described with respect to the foregoing figures, the imaging system 50 can classify the type of occlusion as moisture-related or light-related, and the signal transmitted from the imaging system 50 to the response control system 54 can depend on the type of occlusion.

[0062] For example, the imaging system 50 can detect excrement on the windshield 18 and transmit an output to cause the response control system 54 to control the pump 74 of the window cleaning system to spray a cleaning fluid onto the windshield 18. In another example, the imaging system 50 detects a splash spray from another vehicle 14 in front of the vehicle 14 on the windshield 18 and, in response, transmits a signal to the response control system 54 to control or transmit a message to the HMI 70 to enable the user to control the vehicle 14 to slow down or adjust the positioning of the vehicle 14. The lighting assembly can also be controlled to illuminate the external area 12 of the vehicle 14 at a specific power level / lighting level and / or turn the headlights 48 on or off in response to detecting a moisture condition. These responses are exemplary and non-limiting, such that the moisture detection system 10 can perform any combination of responses in response to the classification performed by the imaging system 50.

[0063] Still referring to Figure 2, the response control system 54 can monitor feedback from the window cleaning system, the moisture control system, the HMI 70, or other vehicle systems to optimize the response of the moisture detection system 10. For example, when the imaging system 50 classifies the environmental conditions in the external area, the imaging system 50 can transmit an instruction or signal indicating light rain in the external area 12 of the vehicle 14 to the response control system 54. In response to detecting light rain, the response control system 54 can initiate wiper control at a low speed. However, manually controlling the wiper 26 to stop wiping the windshield 18 via the second switch 80 can end the automatic control of the wiper 26. The response control can detect the feedback (e.g., the user manually turns off the wiper 26) and update or optimize future responses when detecting light rain. In this way, the detection and classification of environmental conditions (including moisture conditions) and the response thereto can be enhanced to facilitate optimized responses by the user. It is contemplated that feedback (e.g., manual adjustment of the lighting assembly, manual steering adjustment, braking adjustment, etc.) can be used to similarly optimize other examples related to other environmental conditions. In some examples, the user's dismissal of a message presented at the HMI 70 can indicate that the response indicated in the message is not welcome by the user. Other examples of feedback and control will be described with reference to the previous figures.

[0064] Now refer to Figures 3A to 4 and further refer to Figure 2 , according to some aspects, the vehicle 14 includes a camera configured to capture images through a window (e.g., the windshield 18) of the vehicle 14. The window cleaning system can operate between a cleaning operation and a cleaning operation. The window cleaning system includes one or more of the wipers configured to move along the window in the cleaning operation and the cleaning operation. The window cleaning system also includes an interface (e.g., at least one switch 80) for manually activating the cleaning operation. The control circuit is configured to detect an obstruction on the window based on the image; classify the obstruction as water or debris; detect a manual operation of the wiper 26; determine a wear condition during the manual operation of the wiper 26 and classify the obstruction as debris; and transmit a signal indicating the wear condition.

[0065] Generally, the window cleaning system can operate in an automatic mode and a manual mode. In the automatic mode, the cleaning system works in cooperation with a control circuit (e.g., the image processor 88, the controller 56, the controller 72, and / or other control circuits), the imaging system 50, and / or the distance detection system 52 to operate the wiper 26, the nozzle 28, the valve 76, the pump 74, the motor 78, etc. in response to detecting an obstruction on the windshield 18 identified by the imaging device 20. For example, the imaging system 50 can detect water droplets 108, stains, or other moisture conditions, dust ( Figure 3A ) or other dirt, biological residues (such as insect splashes ( Figure 3B)) or any other debris or obscuration on the windshield 18. Thus, the image processor 88 can process the image 86 and apply the FCDDNN 92 or another neural network to classify the obscuration. Based on the classification of the obscuration, a specific operation to be performed by the window cleaning system is determined. For example, a wiping operation can be determined in response to a moisture condition, and a cleaning operation can be determined in response to a non-moisture obscuration. Such operations can be automatically performed in an automatic mode, similar to the operation of automatically activating headlights in response to dark conditions. In this way, manual intervention may be restricted.

[0066] In the cleaning operation, the sprayer for the cleaning fluid is controlled to apply the cleaning fluid to the windshield 18. For example, the controller 56 controls the pump 74 and / or the valve 76 to output the cleaning fluid to the windshield 18, while (or slightly earlier) transmitting a signal to one or more of the motors 78 to drive the wiper 26 to rotate on the windshield 18. The cleaning fluid can operate to reduce the adhesion of the obscuration to the windshield 18 to allow the wiper 26 to clear the obscuration. The cleaning operation may not include the activation of the spray and only involves the operation of the wiper 26.

[0067] It is contemplated that in addition to choosing between the cleaning operation and the cleaning operation, the window cleaning system can also control the speed of the wiper 26 (e.g., the RPM of the motor 78), the timing of the wiper 26 (e.g., initially fast, then slower, the applied duration, etc.), the speed or distribution of the cleaning fluid, or other more specific aspects of the window cleaning system. For example, the imaging system 50 can more specifically classify the condition of the windshield 18 based on the level of the obscuration passing through the windshield 18 or the position of the obscuration, such as "very dirty", "dirty", "lightly dirty", etc. In one example, the imaging system 50 detects an obscuration on only one part of the windshield 18 and only activates the corresponding part of the window cleaning system to clear the obscuration (e.g., only one wiper 26, one of the nozzles 28, etc.).

[0068] The window cleaning system can also be or alternatively be manually controlled by a user via an interface (such as switches 82, 84). For example, switches 82, 84 can be knobs on a control lever that, when twisted, initiate a cleaning operation or a clearing operation according to the activated switch 82, 84. The manual activation or deactivation of switches 82, 84 can be monitored by the window cleaning system. When a change in operation is detected, the controller 56 can, for example, update the algorithm to optimize the timing and conditions for the activation or deactivation of the wiper 26 and / or the sprayer. For example, if, in the automatic mode, the wiper 26 operates automatically in response to moisture on the windshield 18, but the user manually deactivates the wiper 26, the response control system 54 can be updated based on this manual feedback to delay the activation of the wiper 26 in the automatic mode for a longer time. In another example, the elapsed usage time of the cleaning or clearing operation (e.g., until the user manually turns off the cleaning or clearing operation) can be tracked to optimize the interval for operating the window cleaning system. Thus, the system 10 of the present disclosure can utilize learning methods, such as those described with respect to the training module 94, to achieve an optimized response.

[0069] The controller 56 or another part of the control circuit can include a database or other memory 91 (e.g., memory 91) that stores wear information including the usage time of the wiper 26. For example, the elapsed usage time of the wiper 26 since the replacement of the wiper blade can be tracked and compared with the elapsed usage time of the wiper 26 used under wear conditions. By way of example, if the user manually activates the clearing operation when there are dry or non-moisture particles on the windshield 18 (as detected by the imaging system 50), the system 10 can track the duration of use of the wiper 26 in such a wear state. For example, the wiper blade of the wiper 26 can be made of natural or synthetic polyisoprene, butadiene, ethylene-propylene-diene rubber, neoprene, or blends thereof, which may wear over time and especially wear faster when applied to a dry or rough surface, where solvents can be used to limit the wear of the blade. Thus, the wiper 26 can wear in the form of wear of the wiper blade, and the motor 78 can be over-twisted due to resistance caused, for example, by the wiper 26 engaging debris. The wear condition can be communicated to the user via a notification device, such as the HMI 70, an audible speaker, a light indicator, or any other visual or audible notification.

[0070] As will be described herein with respect to Figures 6A to 9BFurther described, the imaging system 50 can classify the moisture condition on the windshield 18 based on the level of optical distortion or occlusion. For example, the imaging system 50 can utilize the image processor 88 to distinguish the moisture condition and non-moisture condition on the windshield 18 based on the transparency level such as opacity, color, etc. Thus, the droplets 108, smudges, etc. of the moisture condition can be distinguished from debris.

[0071] Now referring to Figure 4 , an exemplary method 400 for an automatic cleaning mode of a window cleaning system includes enabling the automatic mode at step S402. In the automatic mode, the system 10 recursively classifies the obstructions on the windshield 18 at S404, including the level of the obstruction and the type of the obstruction. When an obstruction is detected (e.g., wiper = yes), the window cleaning system activates the wiper 26 at S406. Simultaneously with or in temporal proximity to the detection of the obstruction, the obstruction is classified as moisture-based or non-moisture-based (e.g., mainly debris), and the system 10 determines at step S408 whether a cleaning fluid should be applied. For example, mainly dry debris may require a cleaning operation, and at step S410, the sprayer sprays the cleaning fluid on the windshield 18 to loosen the debris and allow the wiper 26 to clean the debris. It is contemplated that the method 400 presented herein is merely exemplary, and other modifications (e.g., manual interruption) can limit the automatic mode, and as previously described, the timing and / or classification criteria can be adjusted based on manual feedback to optimize the response of the window cleaning operation.

[0072] The wear condition can be determined based on functions using cycles and / or environmental conditions such as external climate, air quality, etc. For example, the system 10 can include one or more temperature sensors, humidity sensors, wind speed sensors, wind direction sensors, or other weather detection sensors that can detect environmental conditions. The system 10 can also use the geospatial location (via, for example, GPS discussed below) to determine the environmental condition of the vehicle 14. By way of example, the vehicle 14 may be mainly used in a geographical area having one or more common environmental conditions. A vehicle 14 used in a hot and dry climate may be subject to dust conditions, which typically result in sand on the windshield 18 (e.g., windscreen), similar to Figure 3AThe dust shown. Using the wiper 26 during such conditions may result in greater wear of the wiper 26 compared to using the wiper 26 during a moisture condition. In another example, using the wiper 26 during cold or icy conditions may result in greater wear than typical moisture conditions. The control circuit can determine the operation of the vehicle 14 in snowy climates and / or in northern geographical regions where salt is typically applied to roads. Classifying the environment as including road salt may affect the wear condition estimate. For example, since salt mixes with moisture to provide brine, road salt may enhance the wear of the wiper blade, which may increase the wear of the wiper 26. Other conditions, such as muddy conditions (e.g., a combination of a moisture condition and a dirty condition) can affect the wear condition. Thus, the control circuit can update the wear condition in response to the environmental conditions.

[0073] The system 10 may also or alternatively provide multiple wear classifications to the response control system 54. Some environmental conditions may be more likely to cause wear or a greater degree of wear of the wiper 26 compared to other environmental conditions. Thus, the interaction between the wiper 26 and the windshield 18 can be classified by the control circuit. For example, the imaging system 50 can detect the visual clarity through the windshield 18 before and after a window cleaning operation. In a first classification, the wear classification is minimized while allowing sub-optimal cleaning of the windshield. For example, when there is insect residue on the windshield 18, the imaging system 50 can determine sufficient moisture conditions for a wiping operation without applying a spray. However, the water streaks can be monitored during the operation. Thus, the control circuit can transmit an instruction to warn the user (visually or audibly) of sub-optimal cleaning and recommend (or automatically initiate) a spray in response to the water streaks. In this classification, the wear is classified as minimal wear, and the wear condition is minimized.

[0074] In another example, the system 10 classifies the interaction as a second wear classification in which the result of the cleaning operation performed results in increased wear on the wiper 26. For example, when sand is detected on the windshield 18 (similar to Figure 3BWhen there is dust (in the case of dry sand), the interaction between the windshield wiper 26 and the dry sand without spraying may cause significant friction between the wiper blade and the windshield. The system 10 can detect scraping noises and the like via a microphone. Based on the classification of the obstruction and / or the audible noise from the interaction with the windshield wiper 26, the control circuit can classify the wear and update the wear condition accordingly. In addition, the control circuit can automatically activate the sprayer in response to the scraping, or otherwise convey instructions to the user to activate the application of the cleaning fluid. In yet another example, a third wear classification can be determined by the control circuit. The third wear classification can correspond to an interaction that causes at least a portion of the wiper blade of the windshield wiper 26 to break away from or be lost from the body of the windshield wiper 26. Such classification can be detected by image / video analysis via the imaging system 50 of the windshield 18 and thus the windshield wiper 26. For example, the imaging system 50 can classify a sluggish wiper blade.

[0075] Other categories can be determined. Thus, each interaction can be weighted more or less to affect the calculation of the condition of the windshield wiper 26. The control circuit can take into account the external temperature, humidity, wind speed, or other environmental conditions. For example, in freezing conditions (temperatures near or around 0 degrees Celsius), an obstruction on the windshield, although visually presenting as a typical moisture condition, can be determined to be a solid ice condition. If the windshield wiper 26 is still activated, the obstruction may still be present since the obstruction is solid ice and has not been removed by the cleaning operation. In such an example, the control circuit can classify the interaction as significantly increasing the wear condition of the windshield wiper 26. The control circuit can be configured to transmit a signal to indicate to the user that the windshield wiper 26 should not be operated in such conditions. In another example, the control circuit interrupts the operation of the windshield wiper 26. Thus, the classification step in step 404 of method 400 can include the various wear classifications presented and resulting in different responses by the response control system 54.

[0076] It is contemplated that each wear classification can correspond to each estimated modification amount, variable, or multiplier applied to the wear condition. For example, the control circuit can modify the calculation or algorithm used to determine the wear condition (e.g., multiply the normal wear condition by two, three, ten, etc.). In this way, the normal wear of the wiper blade of the windshield wiper 26 may be increased. Thus, the control circuit can re - estimate the life of the windshield wiper 26. For example, using the windshield wiper 26 when there are large modification amounts (e.g., very cold temperatures with ice, very hot temperatures on the windshield 18, etc.) may result in a significant shortening of the life of the windshield wiper 26.

[0077] Now refer to Figure 5, shows an exemplary arrangement of the imaging device 20 relative to the windshield 18 to demonstrate the proximity of the imaging device 20 to the windshield 18 and the enhanced optical and spatial properties of the arrangement. Although shown as being disposed adjacent the upper portion 96 of the windshield 18, it is contemplated that the imaging device 20 may be disposed along any portion of any window of the vehicle 14, which allows the imaging device 20 to capture a view of the area 12 external to the vehicle 14. Generally, the software employed by the moisture detection system 10 of the present disclosure allows the imaging device 20 to be positioned at least partially independently of distance between the lens 98 of the imaging device 20 and the window. For example, and as will be described with respect to Figure 6A and Figure 6B , depending on the distance of the imaging device 20 relative to the windshield 18, water (e.g., condensate, liquid water, frost) on the windshield 18 may appear different in the image. For example, when the imaging device 20 is spaced farther from the windshield 18 relative to abutting or very close (e.g., between 0 and 35 mm) to the windshield 18, the water droplets 108 may appear as discrete circles, dots, or other geometric shapes more clearly defined along the edges 110 of the shape. The processes and methods performed by the moisture detection system 10 of the present disclosure may allow detection of moisture conditions at either or both of a first position 102 (e.g., at a first distance) of the imaging system 50 and a second position 104 (e.g., at a second distance less than the first distance) of the imaging system 20. For example, when the imaging device 20 is at a first distance from the windshield 18, the moisture on the windshield 18 may appear as a blurred water pattern or a less distinct optical distortion because the imaging device 20 is closer to the water on the windshield 18 than when positioned at the second distance. By providing a distance-independent arrangement, the moisture detection system 10 allows for enhanced spacing within the cabin, a larger and more detailed field of view of the imaging device, and a general application that allows the imaging device to be at different distances from the windshield 18.

[0078] Continuing to refer to Figure 5, the windshield 18 may extend at an inclination angle 106 relative to a vertical and / or horizontal orientation. For example, relative to the driving surface 44, the windshield 18 may be tilted upward. The inclination angle 106 of the windshield 18 may further distort or otherwise affect how the moisture condition is presented in the images captured by the imaging device. In an example where the imaging device 20 is in the second position 104 relative to the first position 102, the distortion may be more apparent. Thus, the challenge of capturing the moisture condition of the external environment of the vehicle 14 from the images captured by the imaging device 20 within the cabin may be greater than or different from the image processing for moisture detection of the images captured by the imaging device outside the cabin. For example, when falling on the windshield 18, the water droplets 108 on the windshield 18 may spread into a more elongated shape than the water droplets 108 on a vertical or more inclined surface (such as the passenger window, the rear window, the cover of the external camera, etc.).

[0079] Now refer to Figure 6A and Figure 6B , which shows the difference in the image 86 captured from the first position 102 of the imaging device 20 relative to the second position 104 in Figure 5 . As shown in Figure 6A (the first position 102), the rainwater on the outer surface 16 of the windshield 18 may appear as droplets 108 having a discrete shape or edge 110, which is more clearly defined than the blur 112 of the water condition shown in Figure 6B (the second position 104). The current algorithms and processes performed by the moisture detection system 10 can detect such optical distortion, although slightly. For example, when the imaging device 20 is positioned away from the windshield 18, the droplet detection algorithm can be adapted to detect the moisture condition, and when the imaging device 20 is positioned closer to the windshield 18 ( Figure 6B ), the imaging system 50 can employ a blur 112 detection algorithm to detect the moisture condition on the windshield 18. Thus, the moisture detection system 10 can provide enhanced spacing within the vehicle 14 by providing the imaging device 20 closer to the windshield 18.

[0080] In some examples, the second position 104 is within 50 mm of the windshield 18. In some examples, the second position 104 is between 0 mm and 35 mm from the windshield 18. In some examples, the second position 104 is positioned between 0 mm and 10 mm from the windshield 18. In each of these examples, the second position 104 is close to the windshield 18 to provide enhanced visibility and spacing within the cabin. In these examples, the first position 102 is farther from the windshield 18 than the second position 104.

[0081] Now refer to Figure 7 , by processing Figure 6BThe captured images are used to generate corresponding output images, showing the FCDDNN92. The FCDDNN 92 is a deep learning model designed for unsupervised anomaly detection in data and operates by learning the basic patterns and structures within the image data to distinguish normal patterns from abnormal patterns. The FCDD can include multiple layers 114, where each layer is a convolutional layer. Since each layer can be convolutional, the FCDDNN 92 can capture spatial dependencies and maintain the spatial structure of the input data.

[0082] In the FCDDN, the input data passes through a series of convolutional layers 114, which extract relevant features at different levels of abstraction. These convolutional layers 114 can be followed by pooling layers 114 to downsample the feature maps and reduce the spatial dimensions of the feature maps. Then, the output of the convolutional layers 114 can be flattened and fed into fully connected layers 114, which perform further feature extraction and map the learned features to an anomaly score. Anomaly detection can be achieved by comparing the calculated anomaly score with a predefined threshold, where a score higher than the threshold indicates an abnormal instance. In this example, the predefined threshold can correspond to the edge continuity of the shapes in the image to detect distortions (e.g., blurriness) caused by moisture. These thresholds can be adjusted actively based on the previously described user feedback (e.g., the user manually activates the windshield wiper 26 and / or the cleaning process, the user manually activates the headlight 48, etc.).

[0083] Now refer to Figures 8A to 8C , an exemplary captured image 86 from the imaging device 20 pointing at the windshield 18 is depicted side by side with the filtered image data (output image 90) indicating the moisture condition on the windshield 18. The imaging system 50 can process the captured image 86 in the image processor 88, including processing the captured image 86 in one or more machine learning models and / or neural networks. The imaging system 50 can employ edge detection techniques, histogram equalization, linear filters, image segmentation, convolution, or any combination of any image processing techniques to detect at least one portion of the captured image having a distortion or an occluder. For example, and referring to Figure 8A and Figure 8B , the imaging system 50 can detect a blur 112 on the windshield 18 corresponding to a wetting condition (e.g., moisture on the windshield 18). In another example, the imaging system 50 detects a non-moisture condition, such as one of the windshield wipers 26 moving on the windshield 18 ( Figure 8C)。In these examples, the detection can be applied to the captured image 86 to associate groups of pixels of the captured image 86 with surrounding pixels in order to classify the image data as moisture-related or non-moisture-related. For example, the imaging system 50 can employ lane detection to identify one or more lane lines that define a plurality of lanes 126. Generally, object classification can be performed by the imaging system 50, such as classifying other vehicles 14, streetlights, trees, or any other object captured in the captured image 86.

[0084] Generally, the previously described Figure 7 FCDDNN 92 can be used to provide a humidity estimate. Based on the humidity estimate, which can be a continuously varying value or a binary wet or non-wet value, the response control system 54 can determine the response of the moisture detection system 10. By employing the respective layers 114 of the FCDDNN 92 in combination with manual feedback from the user (as previously described with respect to Figure 2 ), the techniques employed by the moisture detection system 10 can provide an enhanced response. For example, based on the training of the FCDDNN 92, the windshield wiper 26 can be activated when the user would typically activate the windshield wiper 26 or before the user would typically activate the windshield wiper 26. Additionally, the speed of the windshield wiper 26 and / or the activation of the pump 74 for cleaning the windshield 18 can be optimized based on the moisture level and / or the previous activation of the windshield wiper 26 or the pump 74. The headlights 48 of the vehicle 14 can also or alternatively be activated with a limited false activation. For example, the FCDDNN 92 detects the image data and classifies it as an oncoming headlight 48 and differentiates the oncoming headlight 48 from a moisture condition or a non-moisture condition. Additionally, the FCDDNN 92 can distinguish objects with a distinct moiré effect (e.g., a fence). In some examples, the manual feedback includes the manual operation of the windshield wiper 26 (e.g., via at least one switch 80) and / or the maneuvering of the vehicle 14 away from a splash event 118. Additionally, and as will be described below, adjusting the vehicle 14 to another lane or maintaining its position in the current lane can be manual feedback (e.g., ignoring a lane change suggestion or manually overriding to move into another lane).

[0085] Now referring to Figure 9A and Figure 9B , the identification of at least one splash source 116 of the splash event 118 can be performed by the imaging system 50 of the moisture detection system 10. For example, the splash source 116 can include vehicles 14 and non-vehicles 14. The splash source 116 can be a tire 42, a vehicle body, a bridge, an overpass, construction equipment, a fire hydrant, or any other source. Thus, the imaging system 50 can be configured to classify the source using any of the previously described previous image detection techniques.

[0086] Continuing to refer to Figure 9A andFigure 9B , the imaging system 50 can also or alternatively determine the position of the splash source 116 relative to the vehicle 14. Based on the position of the splash source 116 (e.g., the source lane 126a), the response control system 54 can control any one of the window cleaning system, the lighting system, the HMI 70, and / or the motion control system. For example, the imaging system 50 can locate the splash source 116 in another lane among the plurality of lanes 126 other than the lane of the vehicle 14, and based on the position of the splash source 116 in the other lane, can recommend a route to stay in the current lane of the vehicle 14. Thus, generally, the moisture detection system 10 can control or indicate a message to control the vehicle 14 to adjust lanes based on the position of the splash source 116. In this example, as Figure 9B shown, the imaging system 50 detects a plurality of splash sources 116, and the response control system 54 can recommend one of the plurality of lanes 126 based on the proximity of the vehicle 14 to one or more of the plurality of splash events 118 (e.g., based on the following distance 120 of the vehicle 14 from other vehicles 14, based on the relative position of other vehicles 14 to other splash sources such as potholes). In other words, the moisture detection system 10 can detect the first splash source 116 and the second splash source 116, classify each splash source 116 with a correlation value, and recommend a lane or other position for the vehicle 14 based on the correlation values of the first splash source and the second splash source 116. As will be described below, the moisture detection system 10 can also recommend a target following distance 120, a target speed, etc. in response to the correlation value.

[0087] Specifically referring to Figure 9B , at least one splash zone 122 can be determined or calculated by the imaging system 50 based on the image data from the initial image. For example, any of the previously described prior arts (including the application of the FCDDNN 92) can allow the imaging system 50 to estimate the dimensions of the splash zone 122, such as the depth D, width W, and height H of the splash. The dimensions of the splash zone 122 can be determined based on the following distance 120 from the splash source to the vehicle 14 and the captured image 86. For example, the distance sensors 34, 36, 38 previously described with respect to Figure 1 and Figure 2 can be used to determine the following distance 120, and the foremost position of the splash can be detected based on the image processing of the captured image 86. The difference between the following distance 120 and the front of the splash zone 122 can be calculated by the control circuit to determine the depth D of the splash zone 122. In this way, the distance detection and position detection of the system 10 can provide for determining the origin of the splash, the size of the splash, the likelihood of contacting the water of the splash based on distance and / or speed, etc.

[0088] In this example, the imaging system 50 detects three splash zones 122, each of which has a different size. The size of the splash zone 122 can be caused by the size of the tire 42, the speed of the vehicle 14, the lane in which the vehicle 14 is located (e.g., road roughness, potholes, etc.), or any other factor that may affect the size of the tire splash. For example, an uneven road that may cause puddles, bends in the road (e.g., slopes, turns in the road), etc. may further affect the size of the splash. Based on the size of the splash, the imaging system 50 can determine the following distance 120 from each or any splash source 116 (e.g., other vehicle 14). Generally, the size of the splash, the density of the splash, the duration of the splash, or any other aspect related to the magnitude of the splash can be detected by the imaging system 50 and used to classify the priority or ranking (e.g., correlation value) of the splash event 118 to determine the target lane in the multiple lanes 126, the target following distance 120 of the vehicle 14 relative to other vehicles 14 (e.g., minimum following distance 120), the activation of the window cleaning system, the presentation of a message at the HMI 70, the control of the vehicle 14, or any of the previously described responses. The direction of the splash, which may be affected by wind speed and wind direction, can also be detected to enhance the estimation of the splash zone 122 and further enhance the response determined by the response control system 54. Such wind speed and wind direction can be detected from a weather sensor or a global positioning system (GPS) in communication with the moisture detection system 10.

[0089] The moisture detection system 10 can further or alternatively estimate the density of the moisture condition, such as the density of the splash zone 122 or the density of the rain. For example, based on the amount, distribution, or pattern of the blur 112 or other moisture points detected on the windshield 18, the moisture detection system 10 can estimate the density of the rain and activate the windshield wiper 26 or transmit an indication for the user to operate the windshield wiper 26 in response to the amount, distribution, or pattern exceeding a threshold or matching a target distribution or pattern. For example, if more than 25%, 50%, or 90% of the windshield 18 in the field of view of the imaging device 20 has a blur 112, the imaging system 50 can transmit a signal to the response control system 54 to activate the windshield wiper 26. In other examples, the size of the water droplets 108 and / or the blur 112 can be classified by the FCDDNN 92 and compared with the stored moisture conditions to determine the response of the moisture detection system 10.

[0090] In some examples, the moisture detection system 10 may have access to the location of the vehicle 14 relative to the road type. For example, GPS may provide the location of the vehicle 14, thereby providing the road type (e.g., highway, city driving, etc.). In this way, the number of lanes, direction of movement, construction, and / or mode of operation of the road (e.g., one-way road, interstate highway with a median, highway without a median) can be determined. Accordingly, the moisture detection system 10 may prioritize some lanes over others or further adjust the following distance 120 based on the type of road. For example, since the typical speed of the vehicle 14 on an interstate highway is faster than on another road, the moisture detection system 10 may recommend a "slow lane" among two lanes based on the moisture conditions detected on the interstate highway.

[0091] Although the source lane 126a among the plurality of lanes 126 in the previously described features is shown as having the same traffic direction as another lane among the plurality of lanes 126, in some examples, the plurality of lanes 126 includes lanes having a first traffic direction and lanes having a second traffic direction. In this example, the imaging system 50 is configured to classify the splash source 116 as being in a lane having a first traffic direction and / or classify the splash source 116 as being in a lane having a second traffic direction. For example, the imaging system 50 may detect oncoming headlights 48 in an adjacent lane and, in response to that detection, classify the adjacent lane as having an opposite traffic direction. Accordingly, splashes from oncoming traffic can be compared to splashes from leading traffic, and the moisture detection system 10 can provide an enhanced response determination of the target lane of the vehicle 14 based on either or both of the splash events 118 in the first traffic direction and the second traffic direction. It is also contemplated that the previously described distance detection system 52 may be used to further determine the target lane of the vehicle 14 by tracking the following distance 120 from the vehicle 14 to the leading vehicle 14.

[0092] Now referring Figure 10A and Figure 10B , visual representations of radar mapping are shown as being applied to the images presented in Figure 9A and Figure 9B respectively. In these examples, the density of the cross-hatching corresponds to the density of the splash events 118. The control circuit may generate a density map via the distance detection system 52 and compare such density map to a predefined threshold to classify the splash events 118 by intensity level.

[0093] According to an example of the present disclosure, a detection system for a target vehicle 14 (e.g., a following vehicle 14) includes a camera (e.g., camera 38) that captures an image of an area 12 outside the target vehicle 14. The detection system also includes a radar module (e.g., one or more of radars 34) that scans the area 12 to detect a depth D of a splash event 118 in the area 12. An actuator (such as one or more actuators responsive to a response control system 54) is configured to operate in response to a response signal. A control circuit is configured to determine a distance between the target vehicle 14 and the front of the splash event 118 based on the image, compare the front of the splash event 118 with the depth D to determine the intensity of the splash event 118, and transmit a response signal based on the intensity of the splash event 118.

[0094] The image processing and depth analysis techniques employed by the control circuit for determining the intensity level of the splash event 118 may include any method or component (e.g., a neural network) previously described for moisture detection. For example, the control circuit may employ Doppler effect analysis in conjunction with the radar 34 to detect the density of the splash event 118 and / or the depth of the splash event 118. Density may refer to the ratio of the volume of liquid within the splash event to the volume of space (e.g., air). Pixel analysis (e.g., edge detection, a neural network such as FCDDNN 92) may be used to detect the front of the splash event 118 to determine the position of the water droplets / moisture condition relative to the vehicle 14. The distance between the front of the splash event 118 determined based on the image (e.g., the captured image 86 or the output image 90) and the vehicle 14 and the depth D of the splash event 118 determined based on the scan of the radar 34 may be synthesized at the control circuit. For example, a processor of the distance detection system 52, an imaging system 50, or a controller 56 of the response control system 54 may process the distance information and the depth information to calculate the position of the splash source 116. Additionally, the height H and width W of the splash event 118 may be used to estimate the splash zone 122 with enhanced accuracy as compared to image-based only methods. Thus, by employing the radar 34, an accurate intensity level may be determined.

[0095] Continuing to refer Figure 10A and Figure 10B to, the density of each splash zone 122 may be determined by the control circuit using information from the radar 34 and presented with cross-hatching (i.e., dense cross-hatching corresponding to a high moisture density). Using the density information, the control circuit may estimate / determine the position and identity of the splash source 116. Thus, in addition to calculating the volume of the splash event 118 using the enhanced splash zone 122, the origin of the splash event 118 may be determined based on the density.

[0096] The size and density of the splash event 118 can correspond to the intensity of the splash event 118. For example, a large splash can have a higher intensity level relative to a small splash. It can be based on the speed of the vehicle 14 and / or from the target vehicle 14 to the splash source 116 and / or the origin of the splash (e.g., a puddle, pothole, fire hydrant, or the previously described splash source). For example, the response control system 54 can use splash intensity classification to recommend or actively control the spacing between the following vehicle 14 (e.g., the target vehicle 14) and the leading vehicle 14. For example, the system can use this information to recommend increasing the following distance 120 from the leading vehicle 14 based on the intensity of the splash. In one example, the overtaking condition is determined by the control circuit and transmitted to the driver or another user via the display 68 or another notification device (e.g., an audio instruction). For example, if the driver is considering whether the driver should control the vehicle 14 to overtake the leading vehicle 14 when there is a splash event 118, the system can recommend "overtake" or "do not overtake". In this example, the controller 56 can compare the splash event 118 from the leading vehicle 14 with the splash event 118 in the passing lane caused by another leading vehicle 14 in the adjacent lane.

[0097] Now turning to Figure 11 , an example of using the radar 34 and the camera 38 to perform to recommend an overtaking condition is shown at three moments. At the first moment (t = 1), the target vehicle V1 incorporated with the moisture detection system 10 of the present disclosure measures / tracks the splash event 118 from the vehicle V2 ahead in the current lane L1 and the passing vehicle V3 in the passing lane L2 using the information from the radar 34 and the camera 38, while the passing vehicle V3 is overtaking the vehicle V2 ahead. During this time, the following vehicle V1 optimizes the following distance 120 from the vehicle V2 ahead via the control of one or more actuators of the vehicle system (e.g., braking, motion control, etc.) in response to the intensity level of the vehicle V2 ahead. Alternatively, the system 10 recommends the following distance 120, and the user manually controls the following vehicle V1.

[0098] Based on a previously measured splash event 118 generated by an overtaking vehicle 14 and an estimated speed V3 of the overtaking vehicle (determined using speed detection via radar 34 or other means), the system 10 estimates the front end F of the splash event 118 at a second moment (t = 2). It is contemplated that the speed estimate of the other vehicle 14 can be based on image analysis, comparison with the speed of the following vehicle V1, information from the radar 34, or any other speed detection method. Based on the position of the front end F of the splash event 118 relative to the following vehicle V1, the system 10 controls the following vehicle V1 to move into the passing lane L2 or presents an indication to the following vehicle V1 to move into the passing lane L2 at a third moment (t = 3). For example, when the distance between the front end F and the following vehicle 14 exceeds a target following distance 120 (adjusted for the front end F of the splash event 118) or another threshold distance, the system 10 can indicate that appropriate overtaking conditions exist. Of course, other metrics (such as traffic from the rear or elsewhere and other aspects) can be incorporated into the consideration for approving an overtaking. As described herein, recommending or approving an overtaking is related to moisture condition detection and not related to other factors that may affect the approval of overtaking conditions.

[0099] Generally, using the radar 34 relative to imaging can limit the impact of visual occlusions (sunlight, other lighting, other moisture conditions that occlude the field of view of the camera 34) on depth detection for optimal following distance 120 determination and / or execution. By combining image-based detection with radar-based detection, a more accurate determination of the splash intensity can be tracked to allow for enhanced responses (e.g., activation of the windshield wiper 26, control of the vehicle 14, etc.). For example, the previously described aspects regarding optimal window cleaning or clearing can be further optimized by the system 10 more accurately estimating when the moisture condition on the windshield 18 should be removed (e.g., when the windshield wiper 26 should be energized).

[0100] Now refer to Figure 12, An exemplary process 800 performed by the moisture detection system 10 is shown for use with splash detection and moisture condition detection on the windshield 18. At step 802, the imaging device 20 captures an image of the area 12 outside the vehicle 14. At step 804, the image processor 88 detects an event in the area immediately inside the vehicle 14. For example, the event can be a moisture condition, such as a splash event 118, water on the outer surface 16 of the windshield 18, an object on the windshield 18, or any other event related to visibility obstruction or distortion. At step 806, the event is classified as being associated with the windshield 18 (e.g., on the windshield 18) or spaced apart from the windshield 18. For example, if the event is a splash event 118 related to another vehicle 14 in front of the vehicle 14 rather than water splashing on the windshield 18 itself. It is contemplated that depending on the following distance 120 between the splash source 116 and the vehicle 14, the splash event 118 can be classified as both an on-windshield 18 event and an off-windshield 18 event. If the event is an obstruction on the outer surface 16 of the windshield 18, the image processor 88 can classify the obstruction at step 808. For example, the imaging system 50 can classify the obstruction as debris or water. Based on the classification of the obstruction, at step 810, the moisture detection system 10 determines a response. For example, the response can be to initiate the application of a cleaning fluid on the windshield 18 in the case where the obstruction is debris (such as wet debris), or to activate or adjust the speed of one or more of the windshield wipers 26 on the windshield 18 in the case of a moisture condition. At step 812, an output is transmitted to initiate the response determined at step 810. It is contemplated that the response can alternatively be a more passive response, such as presenting a message at the display 68 to indicate to the user to perform one or more of the functions that can alternatively be performed automatically.

[0101] At step 814, feedback in the form of a manual adjustment or inaction (e.g., the user does not follow the recommendation) is transmitted to the response control system 54 and / or the imaging system 50 to further refine the response determined in future events. For example, if the user is instructed to activate the windshield wiper 26 and the user does not activate the windshield wiper 26, the imaging system 50 can increase the target moisture level for determining windshield wiper activation to a threshold to limit false responses in future calculations. Such a threshold can be the threshold of the previously described FCDDNN 92 or another threshold.

[0102] If the event is not related to the condition of the windshield 18, as determined in step 806, the process can continue at step 816 to determine the location of the splash event 118. For example, using the captured image 86, the imaging system 50 can detect the source and / or location (e.g., the splash lane 126a) of the splash event 118 (e.g., tire splash) caused around the vehicle 14. At step 818, a response is determined by the response control system 54. For example, the response can be to adjust the following distance 120 between the splash source 116 and the vehicle 14 by reducing the speed of the vehicle 14. In other examples, the response includes maneuvering the vehicle 14 into another lane among the plurality of lanes 126. In other examples, the response includes presenting instructions via messaging at the HMI 70 to direct the user to maneuver the vehicle 14, adjust the speed of the vehicle 14, etc. Other examples of the response include adjustments to the window cleaning system, such as activating the windshield wiper 26, adjusting the speed of the windshield wiper 26, activating the pump 74 for applying the cleaning fluid, etc. At step 820, an output can be transmitted to initiate the response. Similar to step 812, at step 822, feedback in the form of an action or inaction by the user to undo the response transmitted by the response control system 54 is returned to the moisture detection system 10 for enhancing response determination in future conditions where a splash event 118 is detected.

[0103] Generally, the moisture detection system 10 of the present invention enhances the response of the vehicle 14 to limit the occlusion and / or distortion of the visibility of the external area 12 of the vehicle 14. The image processing techniques employed by the moisture detection system 10 can enhance the sense of space inside the vehicle 14 by allowing the imaging device 20 to be positioned closer to the windshield 18. Additionally, the image processing techniques employed herein can provide enhanced detection of the moisture conditions on the splash source 116 and / or the outer surface 16 of the windshield 18. Based on the detection of these moisture events, the moisture detection system 10 can provide a fast response time for cleaning the windshield 18 and / or optimizing the maneuvering of the vehicle 14.

[0104] As used herein, the term "and / or" when used in connection with a list of two or more items means that any one of the listed items can be employed alone, or any combination of two or more of the listed items can be employed. For example, if a composition is described as containing components A, B, and / or C, the composition can contain: only A; only B; only C; a combination of A and B; a combination of A and C; a combination of B and C; or a combination of A, B, and C.

[0105] In this document, relational terms such as first and second, top and bottom, etc. are used alone to distinguish one entity or action from another entity or action, and do not necessarily require or imply any actual such relationship or order between such entities or actions. The terms "comprising," "including," or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a series of elements includes not only those elements but may also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprising..." does not preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0106] As used herein, the term "about" means that a quantity, size, formulation, parameter, and other quantities and characteristics are not exact and need not be exact, but may be approximate and / or larger or smaller as required: reflecting tolerances, conversion factors, rounding, measurement errors, etc. and other factors known to those skilled in the art. When the term "about" is used to describe an endpoint of a value or range, the present disclosure should be understood to include the specific value or the recited endpoint. Whether or not the numerical values or endpoints of ranges in this specification recite "about," the endpoints of such numerical values or ranges are intended to include two embodiments: one modified by "about" and one not modified by "about." It should also be understood that each endpoint of a range is significant both in relation to and independent of the other endpoint.

[0107] As used herein, the terms "substantially," "essentially," and variations thereof are intended to indicate that the described feature is equal to or approximately equal to a value or description. For example, a "substantially planar" surface is intended to indicate a planar or approximately planar surface. Additionally, "essentially" is intended to indicate that two values are equal or approximately equal. In some embodiments, "essentially" may indicate that the values are within about 10% of each other, such as within about 5% of each other, or within about 2% of each other.

[0108] Unless expressly indicated to the contrary, as used herein, the terms "the," "a," or "an" mean "at least one" and should not be limited to "only one." Thus, for example, a reference to "a component" includes embodiments having two or more such components unless the context clearly indicates otherwise.

[0109] It should be understood that changes and modifications may be made to the foregoing structure without departing from the concepts of the present disclosure, and it should also be understood that such concepts are intended to be covered by these claims unless the appended claims expressly state otherwise in their language.

[0110] According to the present invention, there is provided a vehicle having: a camera configured to capture an image through a window of the vehicle; a window cleaning system operable between a cleaning operation and a washing operation and including: a wiper configured to move along the window in the cleaning operation and the washing operation; and an interface for manually activating the cleaning operation; and a control circuit configured to: detect an obstruction on the window based on the image; classify the obstruction as water or debris; detect a manual operation of the wiper; determine a wear condition during the manual operation of the wiper and based on classifying the obstruction as debris; and transmit a signal indicating the wear condition.

[0111] According to an embodiment, the window cleaning system includes a sprayer configured to apply a cleaning fluid to the window in the washing operation.

[0112] According to an embodiment, the control circuit includes a memory configured to store wear information including the usage time of the wiper.

[0113] According to an embodiment, the usage time includes the duration of the wear condition.

[0114] According to an embodiment, the window cleaning system is operable in an automatic mode in which the cleaning operation and the washing operation are automatically activated in response to detecting the obstruction.

[0115] According to an embodiment, the window cleaning system includes a pump for pressurizing the cleaning liquid to apply it to the window in the washing operation.

[0116] According to an embodiment, a selection between the cleaning operation and the washing operation in the automatic mode is determined based on classifying the obstruction as water or debris.

[0117] According to an embodiment, the window cleaning system is configured to operate the wiper in response to classifying the obstruction as water and operate the sprayer in response to classifying the obstruction as debris.

[0118] According to an embodiment, the control circuit is configured to control the window cleaning system to adjust from the cleaning operation to the washing operation in response to the wear condition.

[0119] According to an embodiment, the control circuit is configured to classify the obstruction at an optical distortion level, wherein classifying the obstruction as water or debris is based on the optical distortion level.

[0120] According to an embodiment, the control circuit is configured to: capture a subsequent image of the window after the manual operation; and classify the wear condition based on the subsequent image by a modification amount.

[0121] According to an embodiment, the modification amount is a calculated multiplier for the wear condition, and wherein the control circuit is configured to select the multiplier from a plurality of modification amounts corresponding to environmental conditions.

[0122] According to an embodiment, the control circuit is configured to transmit a signal for applying a cleaning fluid to the window in response to the classification of the wear condition by the modification amount.

[0123] According to an embodiment, the invention further features a notification device configured to indicate the wear condition in response to the signal.

[0124] According to the present invention, a vehicle is provided, the vehicle having: a camera configured to capture an image through a window of the vehicle; a window cleaning system operable between a cleaning operation and a washing operation and including: a sprayer configured to apply a cleaning fluid to the window during the cleaning operation; a wiper configured to move along the window during the cleaning operation and the washing operation; and an interface for manually operating the wiper; and a control circuit configured to: detect an obstruction on the window based on the image; classify the obstruction as water or debris; detect a manual operation of the wiper; determine a wear condition during the manual operation of the wiper and based on classifying the obstruction as debris; and transmit a signal indicating the wear condition.

[0125] According to an embodiment, the window cleaning system is operable in an automatic mode in which the cleaning operation and the washing operation are automatically activated in response to detecting the obstruction.

[0126] According to an embodiment, the window cleaning system includes a pump for pressurizing the cleaning fluid for application to the window during the cleaning operation.

[0127] According to an embodiment, a selection between the cleaning operation and the washing operation in the automatic mode is determined based on classifying the obstruction as water or debris.

[0128] According to an embodiment, the window cleaning system is configured to operate the wiper in response to classifying the obstruction as water and operate the sprayer in response to classifying the obstruction as debris.

[0129] According to the present invention, there is provided a vehicle having: a camera configured to capture an image through a window of the vehicle; a window cleaning system operable between a cleaning operation and a cleansing operation and including: a sprayer configured to apply a cleaning fluid to the window during the cleansing operation; a wiper configured to move along the window during the cleaning operation and the cleansing operation; and an interface for manually operating the wiper; a notification device configured to indicate a wear condition of the wiper in response to a signal; and a control circuit configured to: detect an obstruction on the window based on the image; classify the obstruction as water or debris; detect a manual operation of the wiper; determine the wear condition during the manual operation of the wiper and based on classifying the obstruction as debris; and transmit the signal indicating the wear condition.

Claims

1. A vehicle, comprising: A camera configured to capture an image through a window of the vehicle; A window cleaning system capable of operating between a cleaning operation and a cleansing operation and comprising: A wiper configured to move along the window during the cleaning operation and the cleansing operation; and An interface for manually activating the cleaning operation; and A control circuit configured to: Detect an obstruction on the window based on the image; Classify the obstruction as water or debris; Detect a manual operation of the wiper; Determine a wear condition during the manual operation of the wiper and based on classifying the obstruction as debris; and Transmit a signal indicating the wear condition.

2. The vehicle according to claim 1, wherein the window cleaning system includes a sprayer configured to apply a cleaning fluid to the window during the cleansing operation.

3. The vehicle according to any one of claims 1 or 2, wherein the control circuit includes a memory configured to store wear information, the wear information including the usage time of the wiper.

4. The vehicle according to claim 3, wherein the usage time includes the duration of the wear condition.

5. The vehicle according to claim 2, wherein the window cleaning system is capable of operating in an automatic mode, in which the cleaning operation and the cleansing operation are automatically activated in response to detecting the obstruction.

6. The vehicle according to claim 5, wherein the window cleaning system includes a pump for pressurizing the cleaning liquid to apply it to the window during the cleansing operation.

7. The vehicle according to claim 6, wherein the selection between the cleaning operation and the cleansing operation in the automatic mode is determined based on classifying the obstruction as water or debris.

8. The vehicle according to claim 7, wherein the window cleaning system is configured to operate the wiper in response to classifying the obstruction as water and operate the sprayer in response to classifying the obstruction as debris.

9. The vehicle according to claim 8, wherein the control circuit is configured to control the window cleaning system to adjust from the cleaning operation to the cleansing operation in response to the wear condition.

10. The vehicle according to claim 9, wherein the control circuit is configured to classify the obstruction at an optical distortion level, and wherein classifying the obstruction as water or debris is based on the optical distortion level.

11. The vehicle according to claim 10, wherein the control circuit is configured to: Capture a subsequent image of the window after the manual operation; and Classify the wear condition with a modified amount based on the subsequent image.

12. The vehicle according to claim 11, wherein the modified amount is a calculated multiplier for the wear condition, and wherein the control circuit is configured to select the multiplier from a plurality of modified amounts corresponding to environmental conditions.

13. The vehicle according to claim 11, wherein the control circuit is configured to transmit a signal for applying a cleaning fluid to the window in response to classifying the wear condition by the modification amount.

14. The vehicle according to any one of claims 1 to 13, further comprising: a notification device configured to indicate the wear condition in response to the signal.

Images