System and method for detecting presence of droplets on vehicle windshield

Detection of windshield droplets through vehicle camera image processing and AI algorithms, solving the problems of low detection accuracy and efficiency in the prior art, realizing effective wiper control under various weather conditions, and improving vehicle rainwater cleaning ability.

CN119078738APending Publication Date: 2024-12-06FORD GLOBAL TECH LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410671099.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-05
Filing Date
2024-05-28
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The rainwater sensors used to detect windshield droplets in existing vehicles have problems with accuracy and efficiency, especially in complex weather conditions, which makes it difficult to effectively activate and control the wiper.

Method used

Using a system based on vehicle camera capture images, detect droplets on the windshield through image processing and artificial intelligence/machine learning algorithms, including converting from RGB images to binary images, identifying unfocused objects and determining the presence of droplets, and then controlling Wiper movement and speed.

Benefits of technology

It improves the accuracy and efficiency of droplet detection, reduces dependence on conventional rainwater sensing modules, can effectively activate wipers under a variety of weather conditions, and improves the vehicle's rainwater cleaning ability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119078738A_ABST
    Figure CN119078738A_ABST
Patent Text Reader

Abstract

Systems and methods for detecting the presence of droplets on a windshield of a vehicle are provided. A vehicle having a windshield and a camera is disclosed. The camera may be configured to capture a first image of an area around the vehicle through the windshield. The vehicle may also include an image processing module configured to determine a plurality of gradients associated with the plurality of pixels in the first image and determine an unfocused object present in the first image based on the plurality of gradients. The image processing module may also be configured to generate a binary image using the first image in response to determining that the object is not focused. The vehicle may also include a processor configured to determine a droplet presence on the windshield based on the binary image and perform a predefined action in response to determining the droplet presence on the windshield.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to systems and methods for detecting the presence of liquid drops on a vehicle windshield, and more particularly to systems and methods for detecting the presence of liquid drops on a vehicle windshield based on images captured by a vehicle camera. Background Art

[0002] Vehicles typically include a windshield through which a driver or passenger of the vehicle can view the surrounding area. For example, a vehicle may include a front windshield through which a driver of the vehicle can view a road on which the vehicle may be traveling, and a rear windshield through which a driver of the vehicle can view objects near the rear side of the vehicle. The vehicle may additionally include wipers that can clean the windshield when there may be rain, snow, etc. on the windshield.

[0003] Modern vehicles additionally include a dedicated rain sensor or rain sensing module that detects the presence of rain on the windshield. The rain sensor facilitates the vehicle to control the movement of wipers on the windshield. For example, when the rain sensor detects the presence of rain on the windshield, the rain sensor can facilitate the vehicle to automatically activate and control the wiper speed. Summary of the invention

[0004] The present disclosure describes a system and method for detecting the presence of droplets on a vehicle windshield based on an image captured by one or more vehicle cameras. The system can obtain a first image from a front vehicle camera, which can be configured to capture an image of a surrounding area in front of the vehicle through a windshield. The first image can be, for example, a red-green-blue (RGB) image, a grayscale image, a color space image, a thermal image, etc. The system can determine a plurality of gradients associated with a plurality of pixels in the first image. In response to determining the plurality of gradients, the system can determine one or more unfocused image portions or unfocused objects present in the first image. In response to determining the unfocused object, the system can convert the first image into a binary image so that the unfocused object can be highlighted in the binary image. The system can also execute an algorithm based on artificial intelligence (AI) / machine learning (ML) to analyze the binary image and determine the presence or absence of droplets on the windshield. The system can also determine a drop count on the windshield in response to determining the presence of droplets.

[0005] In some aspects, the system may include a wiper control module configured to control movement of a vehicle wiper. The system may activate wiper movement in response to determining that droplets are present on the windshield. The system may also control wiper speed based on the determined drop count.

[0006] In other aspects, the system can be configured to obtain input from additional vehicle cameras, sensors, and / or external servers. Additional vehicle cameras may include, for example, rear-facing cameras, side mirror cameras, and the like. Additional sensors may include, for example, microphones, global positioning system (GPS) receivers, and the like. The system may use input obtained from additional cameras, sensors, and / or external servers to predict the presence of rainfall near the vehicle. When the system predicts that rainfall may be present near the vehicle, the system may determine that droplets are present on the windshield. In some aspects, when the front vehicle camera may not be enabled to capture images through the windshield, the system may use input obtained from additional cameras, sensors, and / or external servers to determine the presence of droplets.

[0007] The present disclosure discloses a system and method for detecting the presence of liquid droplets on a vehicle windshield. The system uses images obtained from a vehicle camera to detect the presence of liquid droplets on the vehicle windshield and control the wiper movement, and thus may not require a conventional rain sensing module to control the wiper movement. The system uses an AI / ML-based algorithm to detect the presence of liquid droplets on the vehicle windshield, thereby improving the accuracy of the prediction of the presence of liquid droplets.

[0008] These and other advantages of the present disclosure are provided in detail herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Detailed description is set forth with reference to the accompanying drawings. The use of the same reference numerals may indicate similar or identical items. Various embodiments may utilize elements and / or components other than those shown in the accompanying drawings, and some elements and / or components may not be present in various embodiments. The elements and / or components in the accompanying drawings are not necessarily drawn to scale. Throughout this disclosure, singular and plural terms may be used interchangeably, depending on the context.

[0010] Figure 1 An exemplary environment is depicted in which techniques and structures for providing the systems and methods disclosed herein may be implemented.

[0011] Figure 2 A block diagram of an exemplary system for detecting the presence of liquid droplets on a windshield according to the present disclosure is depicted.

[0012] Figure 3 Depicted are snapshots of an exemplary vehicle having multiple sensors in accordance with the present disclosure.

[0013] Figure 4A Depicted are snapshots of exemplary RGB images in accordance with the present disclosure.

[0014] Figure 4B Depicted are snapshots of exemplary binary images in accordance with the present disclosure.

[0015] Figure 5 A flow chart is depicted of an exemplary method for detecting the presence of liquid droplets on a windshield according to the present disclosure. DETAILED DESCRIPTION

[0016] The present disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the disclosure are shown and which are not intended to be limiting.

[0017] Figure 1 An exemplary environment 100 is depicted in which techniques and structures for providing the systems and methods disclosed herein may be implemented. Environment 100 may include a vehicle 102, which may take the form of any passenger or commercial vehicle, such as, for example, a sedan, a work vehicle, a crossover vehicle, a truck, a van, a minivan, a taxi, a bus, etc. Vehicle 102 may be a manually driven vehicle and / or may be configured to operate in a partially autonomous mode, and may include any powertrain, such as, for example, a gasoline engine, one or more electrically actuated motors, a hybrid powertrain, etc. Vehicle 102 may travel on road 104 during rain, snow, hail, thunderstorms, etc. Figure 1 shown.

[0018] The vehicle 102 may include a front windshield 106 and a rear windshield (not shown) which may be made of laminated glass or any other similar transparent material. The front windshield 106 and the rear windshield may become wet due to rain, such as Figure 1 Specifically, there may be one or more water droplets on the front windshield 106 and the rear windshield. The vehicle 102 may also include a front wiper 108 (or wiper 108) that may be configured to clean the front windshield 106, and a rear wiper (not shown) that may be configured to clean the rear windshield.

[0019] The vehicle 102 may also include one or more vehicle cameras and sensors (hereinafter referred to as "vehicle cameras") that may be configured to capture images, sound signals, and / or determine surrounding environmental conditions near the vehicle. Figure 2 and Figure 3 Detailed description). The vehicle 102 may be configured to obtain images from a vehicle camera and detect the presence of droplets on the front windshield 106 (and rear windshield) based on the obtained images. For example, the vehicle 102 may obtain images from a vehicle dashboard camera (in Figure 3One or more first images captured by the vehicle instrument camera 302 (shown in FIG. 1 ), the vehicle instrument camera can be configured to capture images of the surrounding area in front of the vehicle 102 through the front windshield 106 (specifically, through the front windshield). The first image can be, for example, a red-green-blue (RGB) image, a grayscale image, a color space image, a thermal image, etc. Throughout the following description, the first image is referred to as an RGB image, however, the other image types described above are also within the scope of the present disclosure. The use of RGB images should not be interpreted as limiting the scope of the present disclosure.

[0020] In response to obtaining an RGB image from the vehicle dashboard camera, the vehicle 102 may convert the RGB image into a binary image to highlight the droplets in the binary image. The vehicle 102 may then detect the presence of droplets on the front windshield 106 (and rear windshield) by analyzing the binary image. Figure 2 The process of converting the RGB image into a binary image and detecting the presence of liquid droplets on the front windshield 106 based on the binary image is described in detail.

[0021] In some aspects, the vehicle 102 can analyze the binary image and detect the presence of droplets by using one or more artificial intelligence (AI) / machine learning (ML) based algorithms (e.g., bagging and boosting decision tree based algorithms, deep learning algorithms, convolutional neural network (CNN) based algorithms, YOLO (Just Look Once) algorithms, etc.).

[0022] In response to detecting the presence of one or more droplets on the front windshield 106, the vehicle 102 can activate the wipers 108 to clean the front windshield 106. In some aspects, the vehicle 102 can additionally determine a drop count on the front windshield 106 based on binary image analysis. The vehicle 102 can control the wiper speed based on the determined drop count.

[0023] In additional aspects, the vehicle 102 can determine the presence of droplets on the front windshield 106 by obtaining input from additional vehicle cameras and sensors. In some aspects, when the vehicle dashboard camera may not be enabled to capture images of the surrounding area via the front windshield, the vehicle 102 can determine the presence of droplets based on inputs from additional vehicle cameras and sensors. In other aspects, even when the vehicle dashboard camera may not be enabled to capture images of the surrounding area via the front windshield, the vehicle 102 can determine the presence of droplets based on inputs from additional vehicle cameras and sensors. In the latter case, the vehicle 102 can determine the presence of droplets based on inputs from additional vehicle cameras and sensors to strengthen or enhance the confidence level of droplet presence detection based on analysis of a binary image (which can be generated by converting an RGB image obtained from the vehicle dashboard camera).

[0024] Examples of additional vehicle cameras and sensors may include, but are not limited to, a front vehicle camera, a rear vehicle camera, a right vehicle rearview mirror camera, a left vehicle rearview mirror camera, an environmental condition detection sensor, etc. As an example, when images from the front vehicle camera and the rear vehicle camera indicate cloudy weather conditions or rainfall, the vehicle 102 may detect the presence of liquid droplets on the front windshield 106 even when the vehicle dashboard camera may not be enabled to capture images of the surrounding area via the front windshield.

[0025] Combine the following Figure 2 Further details of the process of detecting the presence of liquid droplets on the front windshield 106 are described.

[0026] The vehicle 102 and / or vehicle operator implements and / or performs operations as described herein in the present disclosure in accordance with the vehicle owner's manual and safety guidelines.

[0027] Figure 2 A block diagram of an exemplary system 200 for detecting the presence of liquid droplets on a windshield according to the present disclosure is depicted. Figure 2 When you can refer to Figure 3 , Figure 4A and Figure 4B .

[0028] System 200 may include a vehicle 202, a user device 204, and one or more servers 206 that are communicatively coupled to each other via one or more networks 208. Vehicle 202 may be coupled to a Figure 1 The user device 204 may be associated with a vehicle operator (not shown) and may include, but is not limited to, a mobile phone, a laptop, a computer, a tablet, a wearable device, or any other similar device having communication capabilities. The server 206 may be part of a cloud-based computing infrastructure and may be associated with and / or include a telematics service delivery network (SDN) that provides services to the vehicle 202 and other vehicles that may be part of a commercial vehicle fleet ( Figure 2 In other aspects, server 206 can provide real-time weather condition information to vehicle 202 and other vehicles operating as part of a vehicle fleet. The weather condition information can include information associated with the presence, absence, and / or level of rain or snow in a geographic area where vehicle 202 may be located, ambient temperature, etc. In additional aspects, server 206 can store and provide to vehicle 202 an AI / ML-based algorithm that can facilitate vehicle 202 to detect the presence of droplets on the vehicle windshield by analyzing images captured by the vehicle camera.

[0029] The network 208 illustrates an exemplary communication infrastructure in which the connected devices discussed in the various embodiments of the present disclosure may communicate. The network 208 may be and / or include the Internet, a private network, a public network, or other configurations operating using any one or more known communication protocols, such as, for example, Transmission Control Protocol / Internet Protocol (TCP / IP), Wi-Fi, UWB based on the Institute of Electrical and Electronics Engineers (IEEE) standard 802.11, and cellular technologies such as Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), High Speed ​​Packet Access (HSPDA), Long Term Evolution (LTE), Global System for Mobile Communications (GSM), and Fifth Generation (5G), to name a few examples.

[0030] The vehicle 202 may include a plurality of units including, but not limited to, a vehicle computer 210, a vehicle control unit (VCU) 212, and a droplet detection system 214. The VCU 212 may include a plurality of electronic control units (ECUs) 216 arranged to communicate with the vehicle computer 210.

[0031] The user device 204 may be connected to the vehicle computer 210 and / or the droplet detection system 214 via a network 208, which may communicate via one or more wireless connections, and / or the user device may communicate using a near field communication (NFC) protocol, Protocol, Wi-Fi, ultra-wideband (UWB), and other possible data connection and sharing technologies to connect directly with the vehicle 202.

[0032] In some aspects, according to the present disclosure, the vehicle computer 210 and / or the droplet detection system 214 can be installed in the vehicle engine compartment (or elsewhere in the vehicle 202). In addition, the vehicle computer 210 can operate as a functional part of the droplet detection system 214. The vehicle computer 210 can be or include an electronic vehicle controller having one or more processors 218 and a memory 220. In addition, the droplet detection system 214 can be separate from the vehicle computer 210 (e.g., Figure 2 ), or may be integrated as part of the vehicle computer 210.

[0033] The processor 218 may be configured to communicate with one or more memory devices (eg, memory 220 and / or memory 224) configured to communicate with a corresponding computing system. Figure 2The processor 218 may utilize the memory 220 to store programs and / or store data in code form to perform various aspects of the present disclosure. The memory 220 may be a non-transitory computer-readable memory that stores droplet detection program code. The memory 220 may include any one or combination of volatile memory elements (e.g., dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc.), and may include any one or more non-volatile memory elements (e.g., erasable programmable read-only memory (EPROM), flash memory, electronically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), etc.).

[0034] According to some aspects, the VCU 212 can share a power bus with the vehicle computer 210 and can be configured and / or programmed to communicate between the systems of the vehicle 202, connected servers (e.g., server 206), and other vehicles operating as part of a vehicle fleet (e.g., server 206). Figure 2 202). The VCU 212 may include or communicate with any combination of ECUs 216, such as, for example, a body control module (BCM) 222, an engine control module (ECM) 224, a transmission control module (TCM) 226, a telematics control unit (TCU) 228, a driver assistance technology (DAT) controller 230, etc. The VCU 212 may also include and / or communicate with a vehicle perception system (VPS) 232, which connects and / or controls one or more vehicle sensing systems 234. The vehicle sensing system 234 may include one or more vehicle sensors, including, but not limited to, radio detection and ranging (RADAR or "radar") sensors configured to detect and locate objects inside and outside the vehicle 202 using radio waves, seating area lock sensors, seating area sensors, light detection and ranging (LiDAR or "laser radar") sensors, door sensors, proximity sensors, temperature sensors, wheel sensors, ambient weather sensors, vehicle wheel sensors, vehicle interior and exterior cameras, etc. Figure 3 An exemplary snapshot / top view of a vehicle 202 having one or more sensors described herein is shown.

[0035] like Figure 3As shown, the vehicle 202 may include a front instrument camera 302. The front instrument camera 302 may be configured to capture images of the surrounding area in front of the vehicle 202 through the front windshield. The vehicle 202 may include additional cameras, including but not limited to a front camera 304a, a rear camera 304b, a left rearview mirror camera and a right rearview mirror camera 304c, a rear instrument camera (not shown), etc. (collectively referred to as additional cameras 306). The vehicle 202 may include Figure 3 Additional cameras not depicted, for example, goal post cameras. Figure 3 The exemplary depiction of additional cameras shown should not be construed as limiting, and the vehicle 202 may include more additional cameras (not shown) without departing from the scope of the present disclosure. The vehicle 202 may also include additional sensors, including but not limited to a built-in microphone 306a, an external microphone 306b, an external rain sensor 306c, a built-in GPS sensor / receiver 306d, an external GPS sensor / receiver 306e, etc. (collectively, additional sensors 306).

[0036] In some aspects, VCU 212 may control aspects of vehicle operation and implement one or more instruction sets received from user device 204 , from one or more instruction sets stored in memory 220 , including instructions to operate as part of droplet detection system 214 .

[0037] The TCU 228 may be configured and / or programmed to provide vehicle connectivity to wireless computing systems on and off the vehicle 202, and may include a navigation (NAV) receiver 236 (which may be the same as the GPS sensor / receivers 306d, 306e) for receiving and processing GPS signals, module (BLEM) 238, a Wi-Fi transceiver, a UWB transceiver, and / or may be configured to communicate between the vehicle 202 and other systems (e.g., a vehicle key fob, Figure 2 Not shown), other wireless transceivers for wireless communication between the computer and the module ( Figure 2 216). The TCU 228 may be configured to communicate with the ECU 216 via a bus.

[0038] ECU 216 may control various aspects of vehicle operation and communications using input from a human driver, input from an autonomous vehicle controller, input from droplet detection system 214, and / or input via wireless signals received over a wireless connection from other connected devices (such as user device 204, server 206, etc.).

[0039] BCM 222 generally includes an integration of sensors, vehicle performance indicators, and varactors associated with vehicle systems, and may include processor-based power distribution circuits that may control functions associated with the vehicle body, such as lights, windows, security devices, cameras, audio systems, speakers, wipers 108, door locks and access controls, and various comfort controls. BCM 222 may also operate as a gateway for bus and network interfaces to communicate with remote ECUs ( Figure 2 (not shown) to interact.

[0040] The DAT controller 230 may provide Level 1 to Level 3 autonomous driving and driver assistance functions, which may include, for example, active park assist, vehicle back-up assist, adaptive cruise control, and / or lane keeping, among other features. The DAT controller 230 may also provide various aspects of user and environmental input that may be used for user authentication.

[0041] In some aspects, the vehicle computer 210 can be connected to an infotainment system 240. The infotainment system 240 can include a touch screen interface portion, and can include a voice recognition feature, a biometric recognition capability that can recognize a user based on facial recognition, voice recognition, fingerprint recognition, or other biometric recognition means. In other aspects, the infotainment system 240 can also be configured to receive user commands via the touch screen interface portion, and / or display notifications, navigation maps, etc. on the touch screen interface portion.

[0042] The vehicle 202 may also include a windshield 242 and a wiper control module 244. The windshield 242 may be the same as the front windshield 106 and / or the rear windshield, and the wiper control module 244 may be part of the BCM 222 or may be separate from the BCM 222 (e.g., Figure 2 The wiper control module 244 may control the operation of the wipers 108 in response to command signals or instructions received from the vehicle computer 210 and / or the droplet detection system 214 .

[0043] The computing system architecture of the vehicle computer 210, VCU 212 and / or droplet detection system 214 may omit certain computing modules. It should be readily understood that Figure 2 The computing environment depicted in FIG. 1 is an example of possible implementations according to the present disclosure and, therefore, should not be considered limiting or exclusive.

[0044] According to some aspects, the droplet detection system 214 can be integrated with and / or implemented as part of the ECU 216. The droplet detection system 214, whether integrated with the vehicle computer 210 or the ECU 216, or operating as a standalone computing system in the vehicle 202, can include a transceiver 246, a processor 248, a computer readable memory 250, and an image processing module 252.

[0045] The transceiver 246 may be configured to receive information / input from one or more external devices or systems (e.g., user device 204, server 206, etc.) via the network 208. For example, the transceiver 246 may receive an AI / ML-based algorithm for droplet detection from the server 206 and / or weather condition information for a geographic location where the vehicle 202 may be located. As another example, the transceiver 246 may receive input or weather condition information from the user device 204. In addition, the transceiver 246 may transmit notifications (e.g., alerts / alarm signals) to the external devices or systems. In addition, the transceiver 246 may be configured to receive notifications (e.g., alarms / alarm signals) from vehicle 202 components (such as the infotainment system 240, the vehicle sensing system 234 (including Figure 3 In addition, the transceiver 246 can transmit notifications (e.g., alarm / warning signals) to vehicle 202 components (such as the infotainment system 240, BCM 222, etc.).

[0046] Processor 248 and memory 250 may be the same or similar to processor 218 and memory 220, respectively. In some aspects, processor 248 may be an AI / ML-based processor that may utilize memory 250 to store programs (e.g., AI / ML-based algorithms obtained from server 206) and / or store data in code form to perform various aspects in accordance with the present disclosure. Memory 250 may be a non-transitory computer-readable memory that stores droplet detection code. In some aspects, memory 250 may additionally store information associated with vehicle 202 and information obtained from vehicle sensing system 234 (e.g., Figure 3 One or more sensing inputs received by the camera and sensor depicted in .

[0047] As described above, the processor 248 can be an AI / ML based processor that can utilize an AI / ML based algorithm stored in the memory 250 to perform one or more tasks. For example, the processor 248 can use an ML based algorithm to detect the presence of droplets in a binary image. The concept of a binary image is described later in the following description. Examples of ML based algorithms include, but are not limited to, algorithms based on bagging and boosting decision trees, algorithms based on deep learning, and the like.

[0048] As will be appreciated by those of ordinary skill in the art, machine learning (ML) is an application of AI whereby the processor 248 may have the ability to automatically learn and enhance based on experience without explicit programming. Machine learning focuses on using data and algorithms to mimic the way humans learn. In some aspects, a machine learning algorithm may be created to perform classification and / or prediction (e.g., predicting the presence of droplets on the windshield 242 based on a binary image).

[0049] Machine learning can be of various types based on the data or signals available to the learning system. For example, machine learning methods can include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Supervised learning is a method that can be supervised by humans. In this method, the machine learning algorithm can use labeled training data and defined variables. In the case of supervised learning, both the input and output of the algorithm can be specified / defined, and the algorithm can be trained to accurately classify the data and / or predict the outcome.

[0050] Generally, supervised learning can be of two types: “regression” and “classification”. In classification learning, a learning algorithm can help to divide a dataset into multiple categories based on different parameters. In this case, a computer program can be trained on a training dataset, and based on the training, the computer program can classify the input data into different categories. Some of the known methods used in classification learning include logistic regression, K-nearest neighbors, support vector machine (SVM), kernel SVM, naive Bayes, decision tree classification, and random forest classification.

[0051] In regression learning, the learning algorithm predicts output values ​​which may be of continuous nature or real valued. Some of the known methods used in regression learning include simple linear regression, multiple linear regression, polynomial regression, support vector regression, decision tree regression, and random forest regression.

[0052] Unsupervised learning is a method that involves algorithms that can be trained on unlabeled data. Unsupervised learning algorithms can analyze data by themselves and find patterns in the input data. Furthermore, semi-supervised learning is a combination of supervised learning and unsupervised learning. Semi-supervised learning algorithms involve labeled training data; however, semi-supervised learning algorithms can still find patterns in the input data. Reinforcement learning is a multi-step or dynamic process. The model is similar to supervised learning, but may not be trained using sample data. The model can "learn as it goes" by using trial and error. A series of successful outcomes can be reinforced to develop the best recommendation or strategy for a given problem in reinforcement learning.

[0053] In an exemplary aspect, the processor 248 can use supervised machine learning techniques to detect the presence of liquid drops on the windshield 242. Specifically, the processor 248 can obtain training / label data from the server 206, and using the training / label data, the processor 248 can detect the presence or absence of liquid drops on the windshield 242 (as a "classification") based on the binary image obtained by the processor 248.

[0054] In other aspects, the processor 248 can detect the presence and / or location of droplets in the binary image by using a neural network model (not shown) that can be stored in the memory 250 and / or obtained from the server 206. The neural network model can be a trained or unsupervised neural network model that can use machine learning to analyze the obtained binary image, which can enable the processor 248 to detect the presence of droplets in the binary image.

[0055] In one or more aspects, the neural network model may include electronic data that may be implemented, for example, as a software component and may rely on a code database, library, script, or other logic or instructions for executing a neural network algorithm by the processor 248. The neural network model may be implemented as code and routines configured to enable a computing device (such as the processor 248) to perform one or more operations (such as detecting the presence and / or location of a droplet in a binary image). In some aspects, the neural network model may be implemented using hardware including a processor, a microprocessor (e.g., for determining the presence of a droplet), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In other aspects, the neural network model may be implemented using a combination of hardware and software.

[0056] Examples of neural network models may include, but are not limited to, deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), CNN-recurrent neural networks (CNN-RNNs), R-CNNs, fast R-CNNs, faster R-CNNs, artificial neural networks (ANNs), RNNs based on long short-term memory (LSTM) networks, CNN+ANNs, LSTM+ANNs, RNNs based on gated recurrent units (GRUs), fully connected neural networks, deep Bayesian neural networks, generative adversarial networks (GANs), and / or combinations of such networks. In some aspects, the neural network model may include numerical computation techniques using data flow graphs. In one or more aspects, the neural network model may be based on a hybrid architecture of multiple deep neural networks (DNNs).

[0057] The image processing module 252 may be configured to process an RGB image (eg, an RGB image obtained from the front instrument camera 302 ) and convert the RGB image into a binary image to highlight unfocused image portions in the RGB image or unfocused objects present in the RGB image, as described below.

[0058] In operation, the image processing module 252 may obtain an RGB image of the surrounding area in front of the vehicle 202 from the front dash camera 302. As described above, the front dash camera 302 may be configured to capture an RGB image through a windshield. Figure 4A An exemplary RGB image 402 captured by the front instrument camera 302 and obtained by the image processing module 252 is shown in FIG.

[0059] like Figure 4A As shown, RGB image 402 may include one or more focused objects 404 (or objects 404) that may be present in front of vehicle 202. Examples of focused objects 404 may include houses, doors, trees, etc. (e.g., Figure 4A RGB image 402 may also include one or more unfocused objects 406 (or objects 406), which may be, for example, water droplets on a windshield (eg, due to rainfall).

[0060] In response to obtaining the RGB image 402 from the front instrument camera 302, the image processing module 252 may determine a plurality of gradients associated with a plurality of pixels in the RGB image 402. It will be appreciated by those skilled in the art that when there may be water droplets (e.g., raindrops) on the windshield, due to the surface tension of the water, a "lens effect" may be induced in the RGB image 402. The lens effect may result in a high local gradient at the edge of the droplet in the RGB image 402.

[0061] In response to determining the plurality of gradients, image processing module 252 may determine or identify object 404 and object 406 in RGB image 402. In some aspects, object 404 may have a corresponding associated gradient that is less than an average gradient value of RGB image 402 (an average gradient value that image processing module 252 may calculate in response to determining the plurality of gradients). Additionally, object 406 may have a corresponding associated gradient that is greater than the average gradient value of RGB image 402. Image processing module 252 may identify object 406 in RGB image 402 by determining the edges of those objects (e.g., water drops) that may have associated gradients that are greater than the average gradient value of RGB image 402.

[0062] In response to identifying the object 406, the image processing module 252 may generate a binary image 408 by using the RGB image 402. Specifically, the image processing module 252 may convert the RGB image 402 into the binary image 408, such as Figure 4B The image processing module 252 may convert the RGB image 402 into a binary image 408 such that the object 406 may be highlighted in the binary image 408. The image processing module 252 may highlight the object 406 such that the gradient associated with the “highlighted” object 406 may be equal to the average gradient value of the RGB image 402, as depicted in the binary image 408.

[0063] The image processing module 252 may then transmit the binary image 408 to the processor 248 (and to the memory 250 for storage purposes). The processor 248 may obtain the binary image 408 from the image processing module 252, and may analyze the binary image 408 to detect the presence of liquid droplets on the windshield 242 based on the analysis. Specifically, the processor 248 may obtain the one or more AI / ML algorithms described above from the memory 250, and execute the instructions included in the algorithm to detect the presence of liquid droplets on the windshield 242 based on the binary image 408.

[0064] In an exemplary aspect, the processor 248 can perform a bagging and boosting decision tree algorithm to detect the presence of droplets in the binary image 408 (and therefore on the windshield 242). The processor 248 can use a training set (which can be stored in the memory 250) that includes a plurality of binary images with droplets and a plurality of binary images without droplets. The processor 248 can use the training set to perform a bagging and boosting decision tree algorithm to classify the binary image 408 as "having droplets" or "not having droplets". Similarly, in alternative or additional aspects, the processor 248 can perform a deep learning algorithm and use the training set to classify the binary image 408 as "having droplets" or "not having droplets".

[0065] In other aspects, the processor 248 may execute the above-described neural network model to detect the presence of droplets on the windshield 242 by using the binary image 408 and detect the droplet locations in the binary image 408. For example, the processor 248 may execute the R-CNN and / or Yolo algorithm and use the training set to detect one or more regions in the binary image 408 with a probability greater than a threshold value of having droplet presence, and then classify each object in the detected region as “droplet” or “no droplet”. In some aspects, the processor 248 may determine / calculate the count of droplets in the binary image 408 based on the detected droplet locations.

[0066] In response to detecting the presence of droplets on the windshield 242 by using the binary image 408, the processor 248 can transmit a first command signal to the wiper control module 244. When the wiper control module 244 receives the first command signal from the processor 248, the wiper control module 244 can activate the wipers 108. In addition, in response to determining the drop count in the binary image 408, the processor 248 can transmit a second command signal to the wiper control module 244. In some aspects, the second command signal can include information associated with the determined drop count. The wiper control module 244 can control the wiper speed based on the drop count in the binary image 408 in response to receiving the second command signal. For example, if the drop count is greater than a first threshold count, the wiper control module 244 can increase the wiper speed, and if the drop count is less than a second threshold count (which can be the same as or different from the first threshold count), the wiper control module can decrease the wiper speed.

[0067] Although the above description describes aspects in which the processor 248 detects / determines the presence of droplets based on an RGB image obtained from the front dash camera 302, in some aspects, the processor 248 may additionally or alternatively determine the presence of droplets on the windshield 242 based on input obtained from the additional camera 304 and / or the additional sensor 306. In this case, the processor 248 may obtain images from the additional camera 304 and may use the above-described AI / ML algorithms (e.g., CNN, deep learning algorithms, etc.) to predict the presence or absence of rain or snow near the vehicle 202. For example, when images captured from the front camera 304a, the rear camera 304b, and / or the left and right rearview mirror cameras 304c indicate the presence of rainfall, the processor 248 may determine that there may be rainfall or snowfall near the vehicle 202. In this case, the processor 248 may determine that there are droplets on the windshield 242 even though the current dash camera 302 may not be enabled to capture images through the windshield and the images obtained from the additional camera 304 indicate the presence of rainfall.

[0068] In some aspects, even when the front dash camera 302 may be enabled to capture images through the windshield, the processor 248 may use images obtained from the additional camera 304 to enhance the droplet presence decision making via binary image analysis and improve the confidence / accuracy of the droplet presence prediction. Specifically, in this case, the processor 248 may correlate the determined presence of rain near the vehicle 202 with the determined presence of droplets on the windshield 242 to ascertain or determine whether to transmit a first command signal to the wiper control module 244. For example, if the processor 248 determines that droplets are present on the windshield 242, but no rain presence is detected near the vehicle 202, the processor 248 may not transmit the first command signal to the wiper control module 244. On the other hand, if the processor 248 determines that droplets are present on the windshield 242 and also detects that rain is present near the vehicle 202 (indicating a higher confidence or probability of rain), the processor 248 may transmit the first command signal to the wiper control module 244. In this case, the processor 248 may execute another AI algorithm and use another training data set to correlate the rain presence information with the droplet presence information to ascertain or determine whether to transmit the first command signal to the wiper control module 244 .

[0069] In yet another aspect, the processor 248 may obtain input from the additional sensor 306 and / or obtain weather condition information from the user device 204, the server 206, and / or other vehicles in the vehicle fleet (e.g., via vehicle-to-vehicle communication) to predict the presence of rainfall near the vehicle 202. For example, when the sound signal obtained from the external microphone 306b and the weather condition information obtained from the user device 204 and / or the server 206 indicate the presence of rainfall, the processor 248 may determine that rainfall may be present near the vehicle 202. Similar to the above aspect, also in this case, the processor 248 may determine that droplets are present on the windshield 242 by predicting the presence of rainfall when the front instrument camera 302 may not be enabled to capture images through the windshield or by enhancing the droplet presence decision via binary image analysis.

[0070] Figure 5 A flow chart depicts an exemplary method 500 for detecting the presence of liquid droplets on the windshield 242 according to the present disclosure. Figure 5 The following process is exemplary and is not limited to the steps described below. Furthermore, alternative embodiments may include more or fewer steps shown or described herein, and may include these steps in an order different from that described in the exemplary embodiments below.

[0071] The method 500 begins at step 502. At step 504, the method 500 may include obtaining, by the processor 248, the binary image 408 from the image processing module 252. As described above, the image processing module 252 may generate the binary image 408 by converting the RGB image 402 obtained from the front instrument camera 302 into the binary image 408. The object 406 may be highlighted in the binary image 408.

[0072] At step 506, the method 500 may include determining, by the processor 248, the presence of liquid drops on the windshield 242 based on the binary image 408. As described above, the processor 248 may determine the presence of liquid drops on the windshield 242 by using one or more AI / ML algorithms.

[0073] At step 508, the method 500 may include executing, by the processor 248, a predefined action based on the droplet presence determination. As one example, in response to determining that droplets are present on the windshield 242, the processor 248 may transmit a first command signal to the wiper control module 244. As described above, the wiper control module 244 may control wiper movement based on the first command signal.

[0074] Method 500 may end at step 510 .

[0075] In the above disclosure, reference has been made to the accompanying drawings that form a part of the above disclosure, which illustrate specific embodiments in which the present disclosure can be practiced. It should be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present disclosure. References to "one embodiment," "embodiment," "example embodiment," etc. in this specification indicate that the described embodiment may include specific features, structures, or characteristics, but each embodiment may not necessarily include the specific features, structures, or characteristics. In addition, such phrases do not necessarily refer to the same embodiment. In addition, when features, structures, or characteristics are described in conjunction with an embodiment, whether or not explicitly described, those skilled in the art will recognize such features, structures, or characteristics in conjunction with other embodiments.

[0076] In addition, where appropriate, the functions described herein may be performed in one or more of the following: hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) may be programmed to perform one or more of the systems and programs described herein. Certain terms are used throughout the specification and claims to refer to specific system components. As will be appreciated by those skilled in the art, components may be referred to by different names. This document is not intended to distinguish between components that have different names but the same function.

[0077] It should also be understood that the word "example" as used herein is intended to be non-exclusive and non-limiting in nature. More specifically, the word "example" used herein indicates one of several examples, and it should be understood that no undue emphasis or preference is placed on the specific example described.

[0078] Computer-readable media (also referred to as processor-readable media) include any non-transitory (e.g., tangible) media that participate in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take many forms, including but not limited to non-volatile media and volatile media. A computing device may include computer-executable instructions, where the instructions are executable by one or more computing devices (such as those listed above) and stored on a computer-readable medium.

[0079] With respect to the processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes may be practiced by performing the described steps in an order different from the order described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating various embodiments and should in no way be construed as limiting the claims.

[0080] Therefore, it should be understood that the above description is intended to be illustrative rather than restrictive. Upon reading the above description, many embodiments and applications other than the examples provided will be apparent. The scope should not be determined with reference to the above description, but should be determined with reference to the entire scope of the appended claims and equivalents to which such claims are entitled. It is anticipated and expected that the technology discussed herein will develop in the future, and the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that the present application is capable of modification and change.

[0081] Unless explicitly indicated to the contrary in this article, all terms used in the claims are intended to be given their ordinary meanings as understood by the skilled person described herein. Specifically, unless the claims state an explicit limitation to the contrary, the use of singular articles such as "one", "the", "said" and the like should be interpreted as describing one or more of the indicated elements. Unless otherwise specifically stated or understood in other ways within the context when used, conditional language such as, in particular, "can", "may", "can" or "may" is generally intended to express that certain embodiments may include certain features, elements and / or steps, while other embodiments may not include certain features, elements and / or steps. Therefore, such conditional language is generally not intended to imply that one or more embodiments require each feature, element and / or step in any way.

[0082] In one aspect of the present invention, the additional vehicle camera includes at least one of the following: a right side rearview mirror camera, a left side rearview mirror camera, a door pillar camera, and a rear camera.

[0083] In one aspect of the invention, determining the presence of droplets on a vehicle windshield includes determining the presence of droplets by using at least one of: bagging and boosting decision tree algorithms, deep learning, and convolutional neural network algorithms.

[0084] According to the present invention, a non-transitory computer-readable storage medium is provided, on which instructions are stored, which, when executed by a processor, cause the processor to: obtain a binary image from an image processing module of a vehicle, wherein the image processing module is configured to: obtain a first image from a vehicle camera configured to capture a first image of an area surrounding the vehicle; determine multiple gradients associated with multiple pixels in the first image; determine an unfocused object present in the first image based on the multiple gradients; generate a binary image using the first image in response to determining the unfocused object, wherein the binary image highlights the unfocused object; and determine the presence of droplets on a windshield of the vehicle based on the binary image; and perform a predefined action based on the determination of the presence of droplets.

[0085] In one aspect of the present invention, the predefined action includes transmitting a first signal to a wiper control module.

[0086] In one aspect of the invention, unfocused objects have associated gradients at object edges that are greater than the average image gradient value.

Claims

1. A vehicle comprising: windshield; a vehicle camera configured to capture a first image of an area surrounding the vehicle; An image processing module, wherein the image processing module is configured to: obtaining the first image from the vehicle camera; determining a plurality of gradients associated with a plurality of pixels in the first image; determining the presence of an unfocused object in the first image based on the plurality of gradients; and generating a binary image using the first image in response to determining the unfocused object, wherein the binary image highlights the unfocused object; as well as a processor communicatively coupled to the image processing module, wherein the processor is configured to: Obtain the binary image from the image processing module; determining the presence of droplets on the windshield based on the binary image; and A predefined action is performed based on the determination of the presence of the droplet. 2 . The vehicle of claim 1 , wherein the vehicle camera captures the first image of the area surrounding the vehicle through the windshield.

3. The vehicle of claim 1 , wherein the predefined action comprises transmitting a first signal to a wiper control module.

4. The vehicle of claim 1, wherein the unfocused objects have associated gradients at object edges that are greater than a mean image gradient value.

5. The vehicle of claim 1 , wherein the processor is further configured to: determining a count of droplets present on the windshield based on the binary image; and A second signal is transmitted to a wiper control module in response to determining the drop count, wherein the second signal includes information associated with the drop count.

6. The vehicle of claim 1 further comprising an additional vehicle camera.

7. The vehicle of claim 6, wherein the additional vehicle cameras include at least one of: a right side mirror camera, a left side mirror camera, a door pillar camera, and a rear camera.

8. The vehicle of claim 6, wherein the processor is further configured to: obtaining input from said additional vehicle camera; determining the presence of rainfall in the vicinity of the vehicle based on the input from the additional vehicle camera; correlating the determined presence of rainfall proximate the vehicle with the determined presence of droplets on the windshield; and The predefined action is performed based on the association.

9. The vehicle of claim 1, wherein the processor determines the presence of the droplets on the windshield based on the binary image by using at least one of: a bagging and boosting decision tree algorithm, a deep learning, and a convolutional neural network algorithm.

10. A method for detecting the presence of liquid droplets on a vehicle windshield, the method comprising: A binary image is obtained by a processor from an image processing module of the vehicle, wherein the image processing module is configured to: obtaining a first image from a vehicle camera, the vehicle camera being configured to capture the first image of an area surrounding the vehicle; determining a plurality of gradients associated with a plurality of pixels in the first image; determining the presence of an unfocused object in the first image based on the plurality of gradients; and generating the binary image using the first image in response to determining the unfocused object, wherein the binary image highlights the unfocused object; determining, by the processor, the presence of droplets on the vehicle windshield based on the binary image; as well as A predefined action is performed by the processor based on the determination of the presence of the droplet.

11. The method of claim 10, wherein the vehicle camera captures the first image of the area surrounding the vehicle through the vehicle windshield. 12 . The method of claim 10 , wherein performing the predefined action comprises transmitting a first signal to a wiper control module.

13. The method of claim 10, wherein the unfocused objects have associated gradients at object edges that are greater than a mean image gradient value.

14. The method of claim 10, further comprising: determining a count of droplets present on the vehicle windshield based on the binary image; as well as A second signal is transmitted to a wiper control module in response to determining the drop count, wherein the second signal includes information associated with the drop count.

15. The method of claim 10, further comprising: Get input from attached vehicle cameras; determining the presence of rainfall in the vicinity of the vehicle based on the input from the additional vehicle camera; correlating the determined presence of rainfall proximate the vehicle with the determined presence of droplets on the windshield; as well as The predefined action is performed based on the association.