Vehicle glass contamination assessment to optimize automatic start of wash system

Abstract

A vehicle includes a system for cleaning contaminants from a vehicle surface. The system includes a camera for obtaining an image of a surface including a contaminant, a plurality of cleaning devices for cleaning the contaminant from the surface, and a processor. The processor is configured to determine a contamination measure from the image, the contamination measure indicating a contamination level of the surface from the image, determine a contamination area and a contamination type from the image, select a cleaning method for cleaning the surface based on the contamination measure, the contamination area, and the contamination type, the cleaning method comprising selecting a cleaning device from a plurality of cleaning devices, selecting a cleaning direction and a cleaning duration, and controlling the cleaning device using the cleaning method.

Description

Technical Field

[0001] The present disclosure relates to cleaning systems in vehicles, and in particular, to a method of automatically selecting an optimal method for cleaning a surface of a vehicle based on an image of the surface. Background Art

[0002] During normal operation, a vehicle accumulates dirt, rain, snow and other pollutants on one or more surfaces (e.g., windshield) thereof. The vehicle typically includes one or more cleaning systems that can be used to clean pollutants from the surface. The cleaning system may include a plurality of cleaning devices that are suitable for different pollutant types. The cleaning system is typically manually operated so that the driver can select appropriate cleaning devices and select applications, etc. In order to automate the cleaning system, it is necessary to be able to make decisions that would otherwise be made by the driver. Therefore, it is desirable to provide a cleaning system that can automatically select the best method for cleaning pollutants from vehicle surfaces. Summary of the invention

[0003] In one exemplary embodiment, a method for cleaning contaminants from a vehicle surface is disclosed. An image of a surface is obtained using a camera. A processor determines a contamination measure from the image, the contamination measure indicating a level of contamination of the surface from the image. The processor determines a contamination area and a contaminant type from the image. The processor selects a cleaning method for cleaning the surface based on the contamination measure, the contamination area, and the contaminant type, the cleaning method comprising selecting a cleaning device from a plurality of cleaning devices, selecting a cleaning direction, and selecting a cleaning duration. The cleaning device is controlled using the cleaning method.

[0004] In addition to one or more features described herein, the method includes using the speed of the vehicle to select a cleaning device, duration, and orientation.

[0005] In addition to one or more features described herein, the method also includes determining a contamination level based on an average size of the contaminants and a dispersion of the contaminants on the surface.

[0006] In addition to one or more features described herein, the method also includes determining the contaminant type and the contamination level based on one of a single image when the vehicle is stationary and images at multiple time intervals when the vehicle is moving.

[0007] In addition to one or more features described herein, the method also includes determining contaminated areas using semantic segmentation of the image.

[0008] In addition to one or more features described herein, the method also includes inputting the image into one of a predictive model and a machine learning model to determine the pollutant type and the contamination level.

[0009] In addition to one or more features described herein, the method includes comparing the image of the surface to a contamination model of the vehicle.

[0010] In another exemplary embodiment, a system for cleaning contaminants from a vehicle surface is disclosed. The system includes a camera for obtaining an image of a surface, the surface including contaminants, a plurality of cleaning devices for cleaning the contaminants from the surface, and a processor. The processor is configured to determine a contamination measure from the image, the contamination measure indicating a level of contamination of the surface from the image, determine a contamination area and a contaminant type from the image, select a cleaning method for cleaning the surface based on the contamination measure, the contamination area, and the contaminant type, the cleaning method including selecting a cleaning device from a plurality of cleaning devices, selecting a cleaning direction, and selecting a cleaning duration, and controlling the cleaning device using the cleaning method.

[0011] In addition to one or more features described herein, the processor is further configured to use the speed of the vehicle to select a cleaning device, duration, and orientation.

[0012] In addition to one or more features described herein, the processor is further configured to determine the contamination level based on an average size of the contaminants and a dispersion of the contaminants on the surface.

[0013] In addition to one or more of the features described herein, the processor is further configured to determine the pollutant type and the pollutant level based on one of the single image when the vehicle is stationary and the plurality of temporally spaced images when the vehicle is moving.

[0014] In addition to one or more features described herein, the processor is further configured to determine the contaminated area using semantic segmentation of the image.

[0015] In addition to one or more features described herein, the processor is further configured to operate one of a predictive model and a machine learning model to determine a pollutant type and a pollutant level based on the image.

[0016] In addition to one or more features described herein, the processor is further configured to compare the image of the surface to a contamination model of the vehicle.

[0017] In yet another exemplary embodiment, a vehicle is disclosed. The vehicle includes a camera for obtaining an image of a surface, the surface including contaminants, a plurality of cleaning devices for cleaning the contaminants from the surface, and a processor. The processor is configured to determine a contamination measure from the image, the contamination measure indicating a level of contamination of the surface from the image, determine a contamination area and a contaminant type from the image, select a cleaning method for cleaning the surface based on the contamination measure, the contamination area, and the contaminant type, the cleaning method including selecting a cleaning device from a plurality of cleaning devices, selecting a cleaning direction, and selecting a cleaning duration, and controlling the cleaning device using the cleaning method.

[0018] In addition to one or more features described herein, the processor is further configured to use the speed of the vehicle to select a cleaning device, duration, and orientation.

[0019] In addition to one or more features described herein, the processor is further configured to determine the contamination level based on an average size of the contaminants and a dispersion of the contaminants on the surface.

[0020] In addition to one or more of the features described herein, the processor is further configured to determine the pollutant type and the pollutant level based on one of the single image when the vehicle is stationary and the plurality of temporally spaced images when the vehicle is moving.

[0021] In addition to one or more features described herein, the processor is further configured to determine the contaminated area using semantic segmentation of the image.

[0022] In addition to one or more features described herein, the processor is further configured to operate one of a predictive model and a machine learning model to determine a pollutant type and a pollutant level based on the image.

[0023] The above features and advantages and other features and advantages of the present disclosure are apparent from the following detailed description when taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Other features, advantages and details appear, by way of example only, from the following detailed description, which refers to the accompanying drawings, in which:

[0025] Figure 1 A vehicle according to an exemplary embodiment is shown;

[0026] Figures 2A-2C A cleaning device that can be used to clean a glass surface in an exemplary embodiment is shown;

[0027] Figure 2D shows a side view of a cleaning apparatus;

[0028] Figure 2E A side view of the cleaning apparatus is shown in operation.

[0029] Figure 3A The effect of vehicle speed on the movement of contaminants or fluids on a glass surface (e.g., a windshield) is shown;

[0030] Figure 3B The cleaning direction of the cleaning device based on the vehicle speed is shown;

[0031] Figure 4Ashows a diagram illustrating a cleaning system in operation when a vehicle is moving at a first vehicle speed in an embodiment;

[0032] Figure 4B shows a diagram illustrating a cleaning system in operation when the vehicle is moving at a second vehicle speed in an embodiment;

[0033] Figure 5 A process flow of a method for cleaning a vehicle surface in an illustrative embodiment is shown; Figure 6 is a flow chart of a method for cleaning a vehicle surface in an embodiment;

[0034] Figure 7 A contamination model that may be used as prior information for determining contamination type in an illustrative embodiment is shown;

[0035] Figure 8 depicts a vision-based clustering process for determining contaminated areas of a surface; and

[0036] Fig. 9 A flow chart of a method for detecting contaminants is shown. DETAILED DESCRIPTION

[0037] The following description is merely exemplary in nature and is not intended to limit the present disclosure, its application, or uses. It should be understood that throughout the drawings, corresponding reference numerals indicate the same or corresponding parts and features.

[0038] According to an exemplary embodiment, Figure 1 A vehicle 100 is shown. The vehicle 100 includes a surface 102. These surfaces can accumulate contaminants or debris, such as fluids, rain, moisture, dirt, mud, etc. Such surfaces can include windshields, windows, etc. The vehicle 100 includes a cleaning system 104 for cleaning the surface 102. The cleaning system 104 includes a plurality of cameras 106, a vehicle speed sensor 108, a controller 110, and one or more cleaning devices 112. Each camera 106 is associated with the surface 102 and is oriented toward its associated surface to be able to obtain an image of the associated surface. The camera 106 can be an internal camera (i.e., located in the cabin of the vehicle 100) or an external camera. A single camera or multiple cameras can be associated with a surface. Each camera 106 communicates with the controller 110 and can send its image to the controller 110. In various embodiments, the camera 106 can send a single image or a sequence of images. The vehicle speed sensor 108 provides a measurement of the vehicle speed to the controller 110.

[0039] One or more cleaning devices 112 include but are not limited to wipers, electrowetting devices, air nozzles, cleaning fluid devices, oscillating devices, heaters, etc. A single cleaning device or multiple cleaning devices can be associated with a surface. Each cleaning device 112 can be activated by a signal from controller 110 to clean contaminants from its associated surface 102.

[0040] The controller 110 may include processing circuitry, which may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated or group) and memory that executes one or more software or firmware programs, a combinational logic circuit, and / or other suitable components that provide the described functions. The controller 110 may include a non-transitory computer-readable medium storing instructions, which, when processed by one or more processors of the controller 110, implements a method for determining the type of pollutant, the location of the pollutant, and the level of contamination on the surface of the vehicle and determining the proximity of the cleaning surface, including selecting a cleaning device, the activation duration of the cleaning device, and the orientation of the cleaning device. Then, according to one or more embodiments detailed herein, the controller 110 may send a signal to activate the selected cleaning device at a selected time and a selected orientation. The cleaning duration may be, for example, 3 seconds (for low), 6 seconds (for medium), and 9 seconds (for high).

[0041] Figures 2A-2C A cleaning device 112 is shown that may be used to clean a glass surface in an exemplary embodiment. Similar cleaning devices are disclosed in US Application Serial No. 17 / 534,180 and US Application Serial No. 17 / 741,740, assigned to General Motors Corporation, the contents of which are incorporated herein by reference in their entirety. Figure 2A A perspective view 200 illustrating components of a cleaning device 112 is shown. The cleaning device 112 includes a multifunctional glass 202 placed over a camera 204. For ease of illustration, the multifunctional glass 202 is shown separated from the camera 204. The multifunctional glass 202 includes a plurality of layers, one of which includes a plurality of electrodes 206. Figure 2B A perspective view 210 is shown with contaminants 212 accumulated on the multifunctional glass 202 . Figure 2C A perspective view 220 illustrating the operation of the cleaning apparatus is shown. The electrodes 206 are activated in a periodic sequence, causing the contaminants 212 to oscillate and move to one side of the multifunctional glass 202.

[0042] Figure 2D A side view 230 of the cleaning device 112 is shown. The multifunctional glass 202 includes several layers, including a substrate 232 inside the vehicle. An electrode layer 234 is on top of the substrate 232. A dielectric layer 236 is on top of the electrode layer 234. A hydrophobic layer 238 is on top of the electrode layer 234 and faces the outside of the vehicle. Contaminants 212 are shown on the hydrophobic layer.

[0043] Figure 2E A side view 240 of the cleaning device 112 in operation is shown. A voltage source 242 applies a periodic voltage to the electrode layer 234 to generate a force on the contaminant 212 that moves the contaminant in a selected direction 244. The voltage source 242 may be operated to move the contaminant in a direction 244. Figure 2E The selected direction 244 shown may move the contaminants in the opposite direction.

[0044] Figure 3A 1 shows the effect of vehicle speed on the movement of contaminants or fluids on a surface 102 (e.g., a windshield). Frame 302 shows the effect of vehicle speed on the movement of contaminants or fluids on a surface 102 (e.g., a windshield). T The first vehicle speed v s The direction of the fluid when moving without any cleaning equipment. Above the velocity threshold v T At vehicle speeds of , the contaminants are carried onto the windshield due to the drag on the droplets, as indicated by the drag arrows 304 .

[0045] Frame 306 shows when the vehicle is moving at a speed less than a threshold speed v T The second vehicle speed v s The direction of the fluid when moving without any cleaning equipment. The second vehicle speed may include the vehicle being stationary or the vehicle moving backward. At this speed, the contaminants on the windshield are naturally transported downward along the windshield by gravity, as shown by gravity arrow 308.

[0046] Figure 3B 310 shows the cleaning direction of the cleaning device based on the vehicle speed. s >v T The cleaning direction is indicated by the cleaning arrow 312. The cleaning arrow 312 is in the same direction as the Figure 3A The drag arrow 304 is in the same direction.

[0047] Frame 314 shows when the vehicle is moving at v s <v T The cleaning direction is indicated by the cleaning arrow 316, which is in the same direction as the cleaning direction. Figure 3A The gravity arrow 308 is in the same direction.

[0048] Figure 4A and Figure 4B The operation of the cleaning system based on vehicle speed is shown. Figure 4A A perspective view 400 of a vehicle windshield 402 is shown, which illustrates the vehicle in an embodiment when the vehicle is traveling at a first vehicle speed (ie, v s >v T) is shown as a cleaning system in operation while moving. Windshield 402 is shown with nozzles 404A-404C at different locations along its perimeter. Each nozzle 404A-404C is configured to spray cleaning fluid onto an associated area 406A-406C of windshield 402. Areas 406A-406C may overlap with each other and may cover the entire windshield 402 or a viewing area of ​​the windshield.

[0049] Nozzles 404A-404C are oriented to spray cleaning fluid onto the windshield with an upward velocity component 409. Thus, the cleaning fluid exerts a force on the contaminants in the same direction that the contaminants were dragged, thereby allowing the contaminants to be quickly and effectively removed from the windshield at its top edge.

[0050] Figure 4B A perspective view 410 of the windshield 402 is shown, which shows that in an embodiment, when the vehicle is at a second vehicle speed (ie, v s <v T ) is moved. Nozzles 404A-404C are oriented to spray cleaning fluid onto the windshield (e.g., onto associated areas 408A-408C) with a downward velocity component 412. Thus, the cleaning fluid exerts a force on the contaminants that allows the contaminants to be quickly and effectively removed from the windshield at the bottom edge of the windshield.

[0051] Notice, Figure 4B The associated regions 408A-408C (of nozzles 404A-404C) are different from Figure 4A The nozzles 404A-404C may be configured to eject cleaning fluid having an upward component or having a downward component based on a signal from a controller. Alternatively, Figure 4A The nozzles 404A-404C include a first set of nozzles oriented to spray cleaning fluid having an upward component along the windshield, and Figure 4B Nozzles 404A-404C comprise a second set of nozzles oriented to spray cleaning fluid having a downward component along the windshield.

[0052] Figure 5 A process flow 500 of a method for cleaning a vehicle surface in an illustrative embodiment is shown. The process flow 500 includes the operation of various modules on a processor, including a detection and characterization module 502, an action mapping module 504, and a cleaning module 506. The detection and characterization module 502 detects the presence of contaminants on a surface and determines the type of contaminant (e.g., dirt, dust, rain, snow, insects, etc.) and the contaminated area of ​​the surface (i.e., the location of the windshield that includes the contaminant) and the level of contamination.

[0053] The detection and characterization module 502 receives inputs from various devices, including one or more images 508 from the camera 106, a pollution model 510 and a pollution threshold 512 from a database. In various embodiments, a prediction model or a machine learning model can be used to identify pollution and determine the pollution level. The image, the pollution model and the pollution threshold can be input to the prediction model or the machine learning model network, which can compare the image with the pollution model to identify the type of pollutant, the pollution level and the pollution area. In various embodiments, the machine learning model is a neural network. The action mapping module 504 receives the vehicle speed 514 from the vehicle speed sensor 108 and receives the pollution type, pollution level and pollution area from the detection and characterization module 502. The action mapping module 504 selects a cleaning method based on these inputs, including one or more cleaning devices, cleaning duration and cleaning direction. The action mapping module 504 sends the selected cleaning method, cleaning duration and cleaning direction to the cleaning module 506, and the cleaning module 506 activates the selected cleaning device within the selected cleaning duration and along the selected cleaning direction.

[0054] Figure 6 600 is a flow chart of a method for cleaning a vehicle surface in an embodiment. In box 602, one or more images are received. One or more images can be used to establish an image quality metric, which can be used for future testing and / or for determining a cleaning method for cleaning a vehicle surface. In box 604, the image is used to establish a reference image, which can be used to define a quality metric later. Exemplary metrics include, but are not limited to, an image quality index (IQI), a structural similarity index measure (SSIM), a variance inflation factor (VIF), a feature similarity index (FSIM), and a peak signal-to-noise ratio (PSNR). The reference image can be an image taken when the vehicle is in a location with good conditions, such as in a garage or other places with good lighting.

[0055] Returning to block 602, the image is sent to block 606. In block 606, the window region with contamination is detected. Alternatively, in block 608, the window bounding box may also be extracted from the three-dimensional geometric model of the vehicle. In block 610, the window bounding box and / or the window region is considered as a region of interest for subsequent analysis.

[0056] In block 612, the quality of the image is characterized for the region of interest. Characterizing the quality may produce an image quality index (IQI). In block 614, if the image quality index is less than a quality threshold (IQI < Q T ), the method returns to block 602, where more images are received. Otherwise, the method proceeds to block 616. In block 616, the image is processed to determine the contaminant type from the image.

[0057] In block 618, the processor performs semantic segmentation on the image to calculate a contamination metric of the surface that quantifies the level of contamination. The contamination metric M may be calculated as shown in equation (1):

[0058] M=ω1×S av +ω2×σ Equation (1)

[0059] Where S av is the average size of the contaminants, σ ​​is the dirt dispersion (e.g., interquartile range), ω1 and ω2 are weights, where

[0060] ω1+ω2=1 Equation (2)

[0061] In block 620, the pollution measure M is compared with the pollution threshold D T A comparison is made to determine the level of contamination. The contamination threshold is a calibrable quantity. In an embodiment, a contamination threshold may be established using a reference image (block 604). For M >= D T , the pollution level is defined as high, and for M<D T , the pollution level is defined as low.

[0062] In block 622, the cleaning method is determined using an action map. The action map receives inputs such as contamination type (from block 616), contamination level (from block 620), and vehicle speed (from block 624), and outputs a cleaning method, including the selected cleaning equipment, activation duration, and equipment orientation. Table 1 summarizes an illustrative action map, including illustrative inputs and illustrative outputs.

[0063] Table 1

[0064]

[0065] In block 626 , the selected cleaning device is controlled or activated using the cleaning method selected using the motion map.

[0066] Figure 7 A pollution model 700 is shown that can be used as a priori information for determining pollution types in an illustrative embodiment. The pollution model can be generated by simulation or by field operation. The pollution model includes a three-dimensional model of a vehicle coated with particles that indicate the deposition location of pollutants through the operation of the vehicle under given conditions (i.e., dirt road, wet road surface, etc.). The particles can indicate the location and thickness of the pollutant layer and other parameters.

[0067] Figure 8A vision-based clustering process 800 for determining contaminated areas of a surface is depicted. In an embodiment, a predictive model or a machine learning model may be used to perform vision-based clustering. In box 802, an image 812 is received. In box 804, semantic segmentation is performed on the image 812 to associate labels or categories with different pixels in the image, as shown in segmented image 814. In box 806, outliers are identified in the segmented image. Outliers may be determined from a feature extraction process. An outlier image 816 shows various outliers. In box 808, a clustering method is performed to generate representative contamination clusters from the outliers, as shown in clustered image 818.

[0068] Fig. 9 A flow chart of a method for detecting contaminants is shown 900. The method includes a multiple image branch 902 and a single image branch 904. The camera can provide images to either branch.

[0069] Multi-image branch 902 involves using multiple images to determine pollutants. Multiple images contain images spaced apart in time from a selected camera. In box 906, a processor extracts a significant area from the image and tracks the movement of the significant area over time. The extraction and tracking process involves using motion information (i.e., wheel speed, steering angle, etc.) from the vehicle, as shown in box 908. In box 910, tracking is used to detect blocked areas. In box 912, a pollution map is generated using a motion-based visual obstruction program. In box 914, the pollution level is determined, and clusters are formed to locate the polluted area. The pollution level (box 916) can be determined based on a first threshold, and the first threshold can be a calibration amount.

[0070] Single image branch 904 involves determining pollutants using a single image. In box 918, a single image is received from a camera. Vehicle speed is not required. In box 920, the image is compared to the pollution model provided in box 922. In box 924, the pollution level is determined and pollution clusters are generated. A second threshold value can be used to determine the pollution level, as shown in box 926. The pollution level can be calibrated.

[0071] In box 928, the outputs from the multi-image branch 902 (contamination levels and clustering) and the outputs from the single image branch 904 (contamination levels and clustering) are fused to obtain a final contamination level and a final clustering map.

[0072] The terms "a" and "an" do not denote a limitation of quantity, but rather denote the presence of at least one of the referenced item. Unless the context clearly indicates otherwise, the terms "or" mean "and / or". References to "aspects" throughout the specification mean that a particular element (e.g., feature, structure, step, or characteristic) described in conjunction with that aspect is included in at least one aspect described herein, and may or may not be present in other aspects. In addition, it should be understood that the described elements may be combined in any suitable manner in the various aspects.

[0073] When an element such as a layer, film, region, or substrate is referred to as being "on" another element, it can be directly on the other element or intervening elements may also be present. In contrast, when an element is referred to as being "directly on" another element, there are no intervening elements present.

[0074] Unless otherwise indicated herein, all test standards are the most current standards in effect as of the filing date of this application or, if priority is claimed, the filing date of the earliest priority application in which the test standards appear.

[0075] Unless defined otherwise, technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0076] Although the above disclosure has been described with reference to exemplary embodiments, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted for its elements without departing from its scope. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the disclosure without departing from the basic scope of the disclosure. Therefore, it is intended that the disclosure is not limited to the specific embodiments disclosed, but will include all embodiments falling within its scope.

Claims

1. A method for cleaning contaminants from a vehicle surface, comprising: obtaining an image of the surface using a camera; determining, at a processor, a contamination metric from the image, the contamination metric indicating a level of contamination of a surface from the image; determining, at the processor, a contaminated area and a contaminant type from the image; selecting, at the processor, a cleaning regime for cleaning the surface based on the contamination measure, the contaminated area, and the contaminant type, the cleaning regime comprising selecting a cleaning device from a plurality of cleaning devices, selecting a cleaning direction, and selecting a cleaning duration; as well as The cleaning mode is used to control the cleaning device. 2 . The method of claim 1 , further comprising using the speed of the vehicle to select the cleaning device, the duration, and the orientation.

3. The method of claim 1 further comprising determining the pollutant type and the pollutant level based on one of: (i) a single image when the vehicle is stationary; and (ii) multiple temporally spaced images when the vehicle is in motion. The method of claim 1 , further comprising determining the contaminated area using semantic segmentation of the image. 5 . The method of claim 1 , further comprising comparing the image of the surface to a contamination model of the vehicle.

6. A system for cleaning contaminants from a vehicle surface, comprising: a camera for obtaining an image of the surface, the surface including the contaminant; a plurality of cleaning devices for cleaning said contaminants from said surface; as well as The processor is configured as: determining a contamination measure from the image, the contamination measure indicating a level of contamination of a surface from the image; determining the contaminated area and the type of contaminant from the image; selecting a cleaning regime for cleaning the surface based on the contamination measure, the contamination area, and the contamination type, the cleaning regime comprising selecting a cleaning device from the plurality of cleaning devices, selecting a cleaning direction, and selecting a cleaning duration; as well as The cleaning device is controlled using the cleaning mode.

7. The system according to claim 6, wherein: The processor is also configured to use the speed of the vehicle to select the cleaning device, the duration, and the orientation.

8. The system according to claim 6, wherein: The processor is also configured to determine the pollutant type and the pollutant level based on one of: (i) a single image when the vehicle is stationary; and (ii) a plurality of temporally spaced images when the vehicle is in motion.

9. The system according to claim 6, wherein: The processor is also configured to determine the contaminated area using semantic segmentation of the image.

10. The system according to claim 6, wherein: The processor is also configured to compare the image of the surface to a contamination model of the vehicle.

Images