Device for detecting water on a surface and method for detecting water on a surface
Through the combination technology of optical sensor and polarization filter, the combined image is generated using absorption and backscattering characteristics, which solves the problem of insufficient accuracy in identifying water on roads in the prior art, and realizes reliable identification and distinction of water, improving the The reliability of safe driving.
Patent Information
- Application Number
- CN202110275829.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-03-13
- Filing Date
- 2021-03-15
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-03-15
AI Technical Summary
The prior art is difficult to reliably identify water on the surface of the road, especially inadequate accuracy under different light and external influences, resulting in the risk of safe driving.
An optical sensor is used to capture images of the surface through different optical bandwidths, combined with a polarization filter for absorption and backscattering characteristics measurements, generate combined images to detect the gathering state of water, and perform pixel-by-pixel analysis through the processor to distinguish between liquid and solid water. .
Reliable identification of water on the road is achieved, interference from external influences is reduced, and the accuracy and reliability of identification is improved. It is suitable for safe driving and autonomous driving systems.
Smart Images

Figure CN113390794B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to a device for detecting water on a surface and to a method for detecting water on a surface. Background Art
[0002] The recognition of road conditions is essential for safe driving. Modern cars estimate the general risk of road conditions with the help of temperature and control sensors. Several principles for actively recognizing water on the road using different key features and results are proposed.
[0003] For example, on an RGB picture it is difficult to distinguish a wet spot from a dirty spot, since both are simply darker. Although water is transparent, it is easily visible in an RGB picture due to several effects that change the path of light. Computer vision algorithms or object detection algorithms can be applied to RGB pictures to locate anomalies on the road (such as puddles, for example). The advantages of this reflection method include the fact that common image sensors can be used, making this method a low-cost solution, and localization and classification are possible. When using RGB cameras, various effects such as polarization and absorption cannot be determined, and therefore the results of the localization and classification methods are highly dependent on the respective situation and are unreliable.
[0004] Another method describes measuring the backscattering characteristics of the surface. The backscattering behavior of a surface is highly dependent on its roughness. Water and ice are smoother than common road materials (such as asphalt). Most of the horizontally polarized part of the light is reflected by the surface of the water or ice, while the vertical part of the light penetrates the water surface and is scattered on the ground and / or ice crystals. Measuring these backscattering characteristics of the environment using a polarization filter provides a reference point for the classification evaluation. Measuring the backscattering characteristics is an inexpensive solution because it allows the use of common image sensors including simple polarization filters. In addition, for example, this solution makes it possible to reliably locate smooth areas such as water and ice. However, water can only be distinguished from ice when a known light source or a polarized light source is used. However, this method is susceptible to external influences, such as changes in external light and / or underground.
[0005] The next method presented is the spectral absorption measurement of water. Water has recognizable absorption characteristics and absorbs much more energy in the range of the infrared spectrum than in the range of visible light. In the near infrared (NIR) range, water shows obvious absorption peaks. When water begins to form crystals or freeze, some of these absorption peaks show a significant shift. For example, the use of specific image sensors (such as image sensors based on specific semiconductors from the III-V group) to increase the spectral sensitivity allows these peaks in the NIR range to be measured, thereby providing a simple method for distinguishing water, ice and snow. The advantage of this method is, for example, a reliable classification between, for example, water, ice, snow and dry soil. The disadvantages of this solution are, for example, expensive hardware, low resolution and poor positioning. Summary of the invention
[0006] It is therefore an object of the present invention to provide a concept for reliably identifying water on a surface.
[0007] This object is achieved by a device for identifying water on a surface as described below or by a method for identifying water on a surface as described below or by a mobile tool as described below or by a computer program as described below.
[0008] In one aspect, the present invention provides a device for identifying water on a surface. The device includes an optical sensor, a processor, and an additional optical sensor. The optical sensor is used to generate a first image of the surface having a first optical bandwidth, and a second image of the surface having a second optical bandwidth, wherein the water has a first absorbance in the first optical bandwidth, and the water has a second absorbance higher than the first absorbance in the second optical bandwidth. The processor is used to combine the first image and the second image to generate a combined image, wherein the surface is reduced or eliminated compared to the water, and to detect the water in the combined image. The additional optical sensor is configured to provide a third image to the processor using a first polarization filter having a first polarization angle, and to provide a fourth image to the processor using a second polarization filter having a second polarization angle, wherein the two polarization filters are different from each other with respect to their polarization angles. The processor is configured to detect the aggregation state of the water when using the third image and the fourth image in the combined image.
[0009] On the other hand, the present invention provides a mobile tool comprising a device for identifying water on a surface as described above and an interface, wherein the interface is configured to warn the driver of the mobile tool and / or affect control of the mobile tool if the device detects a solid state of the water.
[0010] In yet another aspect, the present invention provides a method for distinguishing between liquid and solid aggregate states of water in an area comprising water. The method comprises:
[0011] obtaining a first image of a surface having a first optical bandwidth, obtaining a second image of the surface having a second optical bandwidth, the water having a first absorptivity in the first optical bandwidth, the water having a second absorptivity in the second optical bandwidth that is higher than the first absorptivity,
[0012] combining the first image and the second image to obtain a combined image in which the surface is reduced or eliminated compared to the water,
[0013] detecting said water in said combined image,
[0014] obtaining a third image using a first polarization filter having a first polarization angle, and obtaining a fourth image using a second polarization filter having a second polarization angle, wherein the two polarization filters differ from each other with respect to their polarization angles,
[0015] evaluating picture elements of the third image and the fourth image including areas of water, and
[0016] Based on the evaluation of the picture elements, the aggregation state of the water is detected.
[0017] In yet another aspect, the present invention provides a computer program for executing the method of distinguishing between liquid and solid aggregation states of water in an area comprising water as described above when the computer program is run on a computer.
[0018] An embodiment of the present invention provides a device for identifying water on a surface, the device comprising an optical sensor and a processor.
[0019] The optical sensor is configured to provide a first image of a surface having a first optical bandwidth, and a second image of the surface having a second optical bandwidth, wherein water has a first absorptivity in the first optical bandwidth and water has a second absorptivity higher than the first absorptivity in the second optical bandwidth.
[0020] The processor is configured to combine the first image and the second image to produce a combined image in which the surface is reduced or eliminated compared to the water.
[0021] Furthermore, the processor is configured to detect water in the combined image.
[0022] The example of the invention is based on the finding that relative measurements are superior to absolute measurements in order to reduce the influence of, for example, external influencing factors (such as environment and / or lighting) on the measurement of a physical property. In this case, the physical parameter (e.g., absorption) is extracted from several measurements, for example, performed by using several optical filters. Thus, the device provides a reliable identification of water on a surface.
[0023] To exploit the absorption characteristics of water, the device takes two pictures of the same surface by using an optical sensor that includes bandpass filters for spectral ranges with high and low water absorption, respectively.
[0024] Other materials show a constant absorption rate in a certain spectral range, only water will show a difference. External influences (such as changes in the ground or lighting) will affect both images equally.
[0025] To reduce or eliminate the background, in an embodiment, the processor is configured to combine the images pixel by pixel to provide a combined image. The combined image is a water heat map, with pixels showing a positive probability of water or ice. This may also be referred to as a "confidence".
[0026] Subsequently, water will still be detected in the resulting combined image, for example, when a threshold is used, where pixels with a certain intensity above the threshold are identified as water.
[0027] For example, the device is configured to detect liquid or solid water on a surface such as a road.
[0028] In an embodiment, the first optical bandwidth is selected from the spectral range between 400nm and 900nm, and the second optical bandwidth is selected from the spectral range between 900nm and 1200nm. CMOS sensors or silicon-based image sensors have a sensitivity up to ˜1000nm, or theoretically up to ˜1102nm. A relatively wide spectral range between ˜900nm and ˜1000nm is preferred.
[0029] All aggregation states of water show increased absorbance in the spectral range between 900 nm and 1200 nm compared to the spectral range between 400 nm and 900 nm, while the absorbance of common road materials (such as asphalt) remains negligible in the entire spectral range between 400 nm and 1200 nm compared to the absorption peak of water (at ~980 nm).
[0030] The above mentioned selection of the first optical bandwidth and the second optical bandwidth provides a difference in pixel intensity in areas comprising water between the pictures.
[0031] In an embodiment, the first optical bandwidth has a value between 820nm and 870nm, between 840nm and 880nm, or between 850nm and 890nm at half-width, and the second optical bandwidth has a value between 920nm and 970nm, between 940nm and 980nm, or between 950nm and 990nm at half-width.
[0032] An optimal choice of the first optical bandwidth and the second optical bandwidth provides the largest possible difference in pixel intensity in areas comprising water between pictures.
[0033] In an embodiment, the optical sensor has a vertical polarization filter configured to reduce or eliminate horizontally polarized light.
[0034] As long as the angle of incidence of the rays reflected from the water surface is close to the Brewster angle, the rays reflected from the water surface are mainly horizontally polarized, while the rays that penetrate the water are mainly vertically polarized. The rays that do not penetrate the water are not affected by the absorption of the water. The vertical polarizing filter removes or eliminates the horizontally polarized rays. Since most of the horizontally polarized light experiences specular reflection near the Brewster angle, a vertical polarizing filter can be used to remove the specular reflection.
[0035] In an embodiment, the optical sensor comprises a monochrome sensor or a CMOS sensor or a low-cost silicon sensor. These characteristics are not mutually exclusive. A monochrome sensor does not include a filter layer or includes a filter layer covering the entire area. A monochrome sensor is used when the filter is externally mounted. Once the filter is applied in the form of a different coating, the sensor is no longer monochrome.
[0036] Common, inexpensive silicon-based image sensors are called CMOS sensors. Actually, CMOS is just one type of design. In embodiments, all types of designs can be used, such as CMOS or CCD. However, there are also expensive III-V semimetal-based CMOS sensors. In embodiments, low-cost silicon sensors are used, which have lower sensitivity than more expensive types of sensors. Specific III-V semiconductor sensors that are sensitive in the higher wavelength range of greater than 1100 nm are not necessary.
[0037] Examples of the present invention feature low-cost common CMOS image sensors or less complex monochrome image sensors without Bayer filters.
[0038] In an embodiment, the optical sensor comprises: a first individual sensor comprising a first optical bandpass filter having a first bandwidth; and a second individual sensor comprising a second optical bandpass filter having a second bandwidth; the optical sensor is configured to simultaneously generate a first image with the aid of the first individual sensor and a second image with the aid of the second individual sensor.
[0039] In a further embodiment, the optical sensor includes a third individual sensor having exchangeable optical bandpass filters, the exchangeable optical bandpass filters including a first optical bandpass filter having a first bandwidth and a second optical bandpass filter having a second bandwidth, the optical sensor being configured to sequentially generate a first image with the aid of the third individual sensor including the first optical bandpass filter and to generate a second image with the aid of the third individual sensor including the second optical bandpass filter.
[0040] In an embodiment, the optical sensor includes a fourth individual sensor, which includes a filter layer, which is configured to divide pixels of an image generated by the fourth individual sensor into two groups in rows, columns or in a checkerboard manner, wherein a first optical bandpass filter with a first bandwidth is arranged above the pixels of the first group so that the pixels of the first group generate a first image with the first optical bandwidth, and a second optical bandpass filter with a second bandwidth is arranged above the pixels of the second group so that the pixels of the second group generate a second image with the second optical bandwidth.
[0041] The optical sensor of the device is configured to provide a first image with a first bandwidth and a second image with a second bandwidth as simultaneously as possible, so that the same surface is imaged as much as possible in the first image and in the second image. In an embodiment, these requirements are met by several individual sensors comprising several bandpass filters, by individual sensors comprising exchangeable optical bandpass filters or by individual sensors comprising a filter layer.
[0042] In an embodiment, the processor is configured to use a pixel-by-pixel quotient calculation, a pixel-by-pixel difference calculation, a pixel-by-pixel normalized difference calculation, or a difference calculation between 1 and a quotient calculation when combining the first image and the second image.
[0043] The processor of the device is configured to combine the pictures on a pixel-by-pixel basis when using any of the above calculations to produce a composite image. The combined image is a water heat map. Each pixel of the combined image represents a positive probability value of water or ice in that pixel. This can also be called a "confidence". Conversely, in the case of a simple quotient, it can represent the confidence of a dry surface in the pixel.
[0044] In an alternative embodiment, instead of actively combining the individual pictures, the raw data is passed directly to a trained model, which makes decisions based on the raw data.
[0045] In an embodiment, the processor is configured to provide a folded image. The processor is configured to scan a combined image having pixels on a pixel-by-pixel basis or on a block-by-block basis using a pixel window to calculate a weighted average of pixel values of the pixel window, and create the folded image based on the weighted average, wherein a position of the weighted average in the folded image corresponds to a position of the pixel window in the combined image. This step increases the signal-to-noise ratio (SNR), wherein the signal is the absorption difference and the noise includes all residuals.
[0046] Water absorption rate is about This means that for a water film with a depth of 1 mm and a pixel saturation of 255, it can only absorb a maximum of 2.0·10 -3 ·255=0.51.
[0047] In order to fully measure the absorption effects, neighboring pixels are also combined in a pixel window. The larger the pixel window, the more the absorption effects will be amplified and, therefore, the resulting folded image or water heat map will become more blurred. For example, if a puddle consists of only a single pixel or too few pixels, it will not be recognized.
[0048] In an embodiment, the processor is configured to compare the value of each pixel of the combined image or the folded image derived from the combined image with a threshold value. Pixels having values that show a certain relationship with the threshold value are detected as water. Threshold determination is one of the simplest solutions for water detection. For example, better results can be achieved by "feature detection".
[0049] The combined image, or collapsed image, is a water heat map where pixels represent the probability of the presence of water or ice.
[0050] During the localization or identification of water, the value of the combined image is compared to a threshold value on a pixel-by-pixel basis. That is, for example, a threshold value may be used to make a final decision on a pixel-by-pixel basis or even via "feature extraction" (i.e., identification of continuous areas when using a neural network, for example).
[0051] In an embodiment, the device comprises an additional sensor configured to provide information to the processor.The processor is configured to detect the state of water accumulation when using the information in the combined image.
[0052] Devices including additional sensors enable preventive location and identification of wet areas, puddles and ice or snow accumulation to infer actual hazards or risks of traction loss, a key feature for safe driving and future advanced driver assistance systems (ADAS) or autonomous vehicles. Preventive hazard detection just 0.5 seconds before actual traction loss can enable ADAS to prevent the driver from making erroneous emergency decisions or preemptively deploying airbags, thereby avoiding fatal injuries.
[0053] Absorption measurements alone cannot detect snow and turbid ice. For this reason, the backscatter method provides localization and the absorption measurement is used as confirmation. The sensor thus provides more than just aggregate status information.
[0054] In an embodiment, the additional sensor includes an additional optical sensor. The additional optical sensor is configured to provide a third image using a first polarization filter having a first polarization angle and to provide a fourth image using a second polarization filter having a second polarization angle, the polarization angles of the two polarization filters being different from each other. Furthermore, the additional optical sensor is configured to provide the third image and the fourth image to the processor.
[0055] The combination of absorption and backscatter measurements provides a reliable signature to assess the presence of water or ice on a pixel-by-pixel basis.
[0056] Analysis of the polarization characteristics of the reflection from liquid water shows that the high polarization ratio of light has a mainly horizontal orientation, while ice crystals scatter the light and cause the waves to rotate. The ice crystals produce a more dispersed polarization with a slightly shifted orientation.
[0057] The difference in polarization properties makes it possible to distinguish between water and ice.
[0058] The different polarization planes of the image allow the extraction of backscattering properties, ie the phase, the major axis and the minor axis of the polarization ellipse. The angular offset of the polarization planes is irrelevant here and can be chosen arbitrarily.
[0059] In an embodiment, the additional optical sensor has an individual sensor comprising a polarization filter layer.
[0060] The polarization filter layer is configured to divide the picture elements of the picture generated by the individual sensors into two groups in a row-by-row, column-by-column or checkerboard manner. Above the picture elements of the first group, a first polarization filter is arranged so that the picture elements of the first group generate a third image with a first polarization angle. A second polarization filter is arranged above the picture elements of the second group so that the picture elements of the second group generate a fourth image with a second polarization angle.
[0061] The additional optical sensor comprising a polarization filter layer is configured to generate the third image using the first polarization filter and the fourth image using the second polarization filter as simultaneously as possible so that the same surface is imaged in the pictures as much as possible.
[0062] In an embodiment, the polarization angles of the polarization filters are arranged to be offset from each other by 90°. Two polarization filters with polarization angles of 0° and 90° are sufficient. Three polarization filters with polarization angles offset from each other by 60° provide higher accuracy.
[0063] The uniform distribution of the polarization plane allows for detailed extraction of the major and minor axes of the polarization ellipse, making it easier to distinguish water from ice.
[0064] In an embodiment, the processor is configured to detect an area including water in the combined image or a folded image derived from the combined image. In addition, the processor is configured to evaluate picture elements from the third and fourth images in the area including water and detect the aggregation state of water (i.e., whether the area including water contains water, liquid water or ice).
[0065] The combination of absorption and backscatter measurements provides a reliable signature to assess the presence of water or ice on a pixel-by-pixel basis. Areas containing water are detected by absorption measurements, and the state of water aggregation in the area is assessed by using backscatter measurements.
[0066] In an embodiment, the processor is configured to generate normalized intensity values for the third and fourth images when evaluating picture elements of the third and fourth images.
[0067] Furthermore, the processor is configured to use a comparison of the value of the picture element with the sum of the values of the picture elements of the third and fourth images when generating the normalized intensity value to detect the aggregation state of water.
[0068] The processor is configured to detect that the aggregation state of water is ice when the normalized intensity values of the third and fourth images are within a range of + / - 10% of the mean of the normalized intensity values of the third and fourth images, or to detect that the aggregation state of water is liquid water when the normalized intensity values of the normalized intensity values from the third and fourth images are greater than 1.5 times the mean of the normalized intensity values of the third and fourth images.
[0069] The three images are normalized to the mean of the three images on a pixel-by-pixel basis and compared to each other. Simple thresholding decisions that can be automated can be summarized as follows.
[0070] The ice surface exhibits large scattering, and therefore, the picture elements of all three images will have similar normalized intensity values. If the picture elements of all three images have similar normalized intensity values in the same area, then the picture element in that area will be identified as ice.
[0071] The reflection of liquid water shows a high polarization ratio with light having mainly horizontal orientation. If the normalized intensity in one area of the image is much higher than the intensity of other images in the same area, the picture element in this area will be identified as water.
[0072] In an embodiment, the device comprises a temperature sensor configured to provide an ambient temperature value to the processor, which allows the processor to assess the state of water accumulation.
[0073] The addition of a temperature sensor allows for further complexity reduction. If the ambient temperature is well above or well below the freezing point of water, then an area including water will be identified as either water or ice, depending on the ambient temperature.
[0074] In an embodiment, the processor is configured to sense a set of state values comprising information from the first image and the second image, and to provide a variable threshold value based on the set when using an artificial intelligence element or a neural network. As a variation of "feature detection", a neural network can be used to find a better threshold value on a pixel-by-pixel basis, or to identify continuous shapes in a complete picture.
[0075] Furthermore, the processor is configured to detect water in the combined image or a state of accumulation of water in the combined image when using a variable threshold.
[0076] A variable threshold value created based on information from the first image and from the second image when using artificial intelligence elements or neural networks will lead to better recognition rates in different environmental situations.
[0077] In an embodiment, the processor is configured to sense a set of state values comprising information from the third image and from the fourth image and to provide a variable threshold value as a function of the set when using an artificial intelligence element or a neural network.
[0078] Furthermore, the processor is configured to detect water in the combined image or a state of accumulation of water in the combined image when using a variable threshold.
[0079] A variable threshold created based on information from the third image and the fourth image when using artificial intelligence elements or neural networks will result in better recognition rates in different environmental situations.
[0080] An embodiment of the present invention provides a mobile tool including the device and an interface.
[0081] The device includes additional sensors configured to provide information to the processor.
[0082] The processor of the device is configured to detect the state of water accumulation when using the information in the combined image.
[0083] The interface is configured to alert the driver of the moving tool and / or affect control of the moving tool if the device detects a solid state of water.
[0084] The equipment of the mobile tool enables safe driving through preventive positioning and identification of wet areas, puddles, and ice or snow to infer the actual danger or risk of traction loss. Preventive hazard detection before actual traction loss can enable ADAS to warn the driver via an interface or influence the control of the mobile tool.
[0085] According to a further embodiment of the present invention a corresponding method is provided.
[0086] When using CMOS image sensors, embodiments according to the present invention will demonstrate preventive identification of water and ice hazards on the road. BRIEF DESCRIPTION OF THE DRAWINGS
[0087] Embodiments according to the present invention will be explained in more detail below with reference to the accompanying drawings. With regard to the schematic diagrams shown, it should be noted that the functional blocks shown should be understood as elements or features of the apparatus of the present invention and the corresponding method steps of the method of the present invention, and the corresponding method steps of the method of the present invention can also be understood therefrom.
[0088] Figure 1 A schematic diagram showing an embodiment of a device including an optical sensor and a processor;
[0089] Figure 2 A schematic diagram showing an embodiment of a device including an optical sensor, an additional optical sensor, and a processor;
[0090] Figure 3a A schematic diagram showing an embodiment of a mobile tool including a device and an interface;
[0091] Figure 3b A schematic diagram showing an embodiment of a mobile tool including an apparatus;
[0092] Figure 4a A diagram showing the absorption characteristics of water in the spectral range between 400 nm and 1000 nm;
[0093] Figure 4b A diagram showing the absorption characteristics of water in the spectral range between 400 nm and 1200 nm;
[0094] Figure 5 A schematic diagram showing a method of evaluating an absorption measurement performed by a processor of a device;
[0095] Figure 6 a shows an image without filter over a surface comprising water;
[0096] Figure 6 b shows the combined image above a surface comprising water;
[0097] Figure 7a An unpolarized ray of light partially penetrates the water and partially reflects off the water surface;
[0098] Figure 7b A graph showing the reflectance of perpendicularly and parallel polarized light at different angles of incidence;
[0099] Figure 7cThe reflectance ratio of vertically and parallelly polarized light reflected from the water surface at different incident angles is shown. 's picture;
[0100] Figure 8 a shows three possible polarization ellipses defined by two polarization planes;
[0101] Figure 8 b shows the polarization ellipse defined by the three polarization planes;
[0102] Figure 8 c shows the polarization ellipse of liquid water with a large horizontal polarization component;
[0103] Figure 8 d shows the polarization ellipse of solid water with a uniform polarization part;
[0104] Figure 8 e shows the polarization ellipse of liquid water with unchanged orientation;
[0105] Figure 8 f shows the polarization ellipse of solid water with changed orientation;
[0106] Fig. 9 a shows an image with a first polarization filter above a surface with water;
[0107] Fig. 9 b shows an image with a second polarization filter above the surface with water;
[0108] Fig. 9 c shows an image with a third polarization filter above the surface with water;
[0109] Fig. 9 d shows the image without filter on the surface with water;
[0110] Fig.10 An optical sensor including an image sensor and a filter layer is shown; and
[0111] Fig.11 An embodiment involving three different cameras or photosensitive areas is shown. DETAILED DESCRIPTION
[0112] Before explaining the embodiments of the present invention in detail below with reference to the accompanying drawings, it should be pointed out that exactly the same elements, objects and / or structures with the same function or the same effect are provided with the same or similar figure marks in different drawings, so that the descriptions of these elements given in different embodiments are interchangeable or mutually applicable.
[0113] Figure 1A schematic diagram of an embodiment of an apparatus 100 for identifying water on a surface 150 is shown, the apparatus 100 comprising an optical sensor 110 and a processor 120 coupled to the optical sensor. The output of the processor 120 is the output of the apparatus 100.
[0114] The optical sensor 110 includes a vertical polarization filter 115, a first individual sensor 123 including a first optical bandpass filter 133, and a second individual sensor 126 including a second optical bandpass filter 136. The vertical polarization filter 115 is not required.
[0115] The first optical bandpass filter has a first bandwidth, and water has a first absorptivity in the first bandwidth. The second optical bandpass filter has a second bandwidth, and water has a second absorptivity in the second bandwidth that is higher than the first absorptivity.
[0116] The device is configured to detect water on a surface 150. As will be discussed further below, light 160 partially penetrates the water and partially reflects off the water surface.
[0117] The vertical polarizing filter 115 is configured to reduce or eliminate horizontally polarized rays that are specularly reflected off the water surface.
[0118] The unattenuated radiation will reach the first individual sensor 123 including the first optical bandpass filter 133 or the second individual sensor 126 including the second optical bandpass filter 136 .
[0119] The optical sensor 110 is configured to generate a first image 143 by means of the first individual sensor 123 and a second image 146 by means of the second individual sensor as simultaneously as possible, so that as far as possible the same surface 150 is imaged in the first image and in the second image. Furthermore, the sensor is configured to provide the first image and the second image to a processor.
[0120] The processor is configured to provide a combined image 130 as a combination of the first image and the second image and to detect or locate water in the combined image, as will be discussed further below. The detected area comprising water 140 is an output of the processor or an output of the device.
[0121] Figure 2 A schematic diagram of an embodiment of a device 200 for identifying a state of accumulation of water on a surface 150 by means of an optical sensor 110, an additional optical sensor 210 and a processor 120 coupled to the optical sensor 110 and the additional optical sensor 210 is shown. The output of the processor 120 is the output of the device 200.
[0122] The processor may be coupled to an optional additional temperature sensor 230 .
[0123] The optical sensor 110 includes individual sensors 123, which include a filter layer 233. As indicated in the front view 233F, the filter layer is divided into two groups in a row-by-row, column-by-column or checkerboard manner. A first optical bandpass filter having a first bandwidth is located in the first group, and a second optical bandpass filter having a second bandwidth is located in the second group.
[0124] The first optical bandpass filter has a first bandwidth in which water has a first absorptivity, and the second optical bandpass filter has a second bandwidth in which water has a second absorptivity higher than the first absorptivity.
[0125] The additional optical sensor 210 includes an individual sensor 243, which includes a polarization filter layer 253. Similar to the filter layer 233, the polarization filter layer 253 is divided into three groups in a row-by-row, column-by-column or checkerboard manner. A first polarization filter having a first polarization angle is located in the first group, a second polarization filter having a second polarization angle is located in the second group, and a third polarization filter having a third polarization angle is located in the third group.
[0126] The device 200 is configured to detect a state of water accumulation on the surface 150 .
[0127] The optical sensor 110 is configured to generate a first image 143 having a first bandwidth and a second image 146 having a second bandwidth.
[0128] The additional optical sensor 210 is configured to generate a third image 263 using a first polarization filter, a fourth image 266 using a second polarization filter, and a fifth image 269 using a third polarization filter.
[0129] The optical sensor 110 and the additional optical sensor 210 are configured to provide the first, second, third, fourth and fifth images 143 , 146 , 263 , 266 , 269 to the processor 120 .
[0130] The optional temperature sensor 230 is configured to provide the processor with an ambient temperature value. If the ambient temperature is well above or well below the freezing point of water, then the area containing water will be identified as water or ice, depending on the ambient temperature.
[0131] The processor 120 is configured to evaluate the absorption measurement 280 based on the first and second images and detect water on the surface 150. In addition, the processor 120 is configured to evaluate the backscatter measurement 290 when using the third, fourth and fifth images and detect the concentration state of water in the area including water. The characteristics of the evaluation method 280 of the absorption measurement and the evaluation method 290 of the backscatter measurement will be further explained below.
[0132] The detected aggregation state 240 in the area including water is an output of the processor or an output of the device.
[0133] Figure 3a A schematic diagram of an embodiment of a mobile tool 300 having a device 200 and an interface 310 coupled to the device is shown.
[0134] As in Figure 2 As explained in , the device 200 is configured to detect a gathering state in an area on the surface 150 that includes water.
[0135] The interface 310 is configured to warn the driver 320 of the mobile vehicle and / or preemptively affect control of the mobile vehicle if the device detects a solid state of water.
[0136] The device 200 of the mobile tool 300 enables safe driving by proactively identifying and locating wet areas, puddles, and ice or snow on the road and classifying the accumulation state when the ambient temperature approaches the freezing point of water. For future advanced driver assistance systems (ADAS) or autonomous vehicles, inferring the actual danger or risk of traction loss is a key feature.
[0137] Alternatively, Figure 3b A schematic diagram of a second embodiment of a mobile tool 300 comprising the device 200 is shown. The device 200 comprising an image sensor can look between 5m and 100m forward at a camera height of 1.40m. For example, if the device is to look 100m forward, the angle of incidence 330 is approximately ~89°.
[0138] Figure 4a Graph 400 shows the absorption characteristics 410 of water in the spectral range between 400 nm and 1000 nm. The spectral range between 900 nm and 1000 nm has a high absorbance and is highlighted by window 420 .
[0139] Figure 4b Graph 450 shows the absorption characteristics 410 of water in the spectral range between 400nm and 1200nm. The spectral range between 900nm and 1200nm has two absorption peaks. The device 200 is sensitive in the wavelength range between 900 and 1000nm. A specific III-V semiconductor image sensor that is sensitive in the higher wavelength range >1100nm is not necessary.
[0140] As in Figure 1As explained in , the device 100 utilizes the absorption characteristics of water. The optical sensor 110 of the device 100 takes two pictures 143, 146 on the same surface 150 by means of bandpass filters 133, 136 for spectral ranges with high and low water absorption, respectively. Areas including water are imaged differently in the first image 143 and the second image 146.
[0141] Since, for example, other ambient materials, such as asphalt, show a consistent absorption in the spectral range between 400 nm and 1200 nm, external influencing factors, such as changing background or lighting, will, for example, affect both pictures equally.
[0142] In order to reduce or eliminate the background, or to highlight the area including water, the processor 120 of the device 100 is configured to combine the pictures 143, 146 with each other. By optimally selecting the first optical bandwidth of the first optical bandpass filter 133 and the second optical bandwidth of the second optical bandpass filter 136, the maximum possible difference in pixel intensity in the area including water between the pictures is provided.
[0143] The absorption peak of water is at about 980 nm, so the first bandpass filter is used in the wavelength range between 900 and 1000 nm. In fact, the second bandpass filter can be freely selected in the wavelength range between 400 and 900 nm.
[0144] In the optimal choice, the first optical bandwidth can have a value at half width, for example, between 820nm and 870nm, between 840nm and 880nm, or between 850nm and 890nm, and the second optical bandwidth can have a value at half width, for example, between 920nm and 970nm, between 940nm and 980nm, or between 950nm and 990nm.
[0145] Figure 5 Shown by Figure 2 Schematic diagram of a method 280 for evaluating absorption measurements performed by the processor 120 of the device 200 in FIG.
[0146] As a starting point, a surface 510 is shown that is partially covered by water 520, such as a road with puddles.
[0147] As a first step, the optical sensor 110 of the device 200 generates a first image 143 having a first optical bandwidth, and generates a second image 146 having a second optical bandwidth, in which the water 520 has a first absorbance, and in which the water has a second absorbance higher than the first absorbance.
[0148] Due to the different absorption rates, the area including the water 520 is imaged differently in the first image 143 and the second image 146. In the second image 146, the area including the water 520 is significantly darker.
[0149] As a second step, the processor 120 of the device 200 generates a combined image 530 from the first image 143 and the second image 146. In this case, a pixel-by-pixel quotient calculation is used in the pixel-by-pixel combination. Other methods of combination may include a pixel-by-pixel difference calculation, a pixel-by-pixel normalized difference calculation, or a difference calculation between 1 and a quotient calculation.
[0150] The result of the combined images or pixel-by-pixel quotient calculation is a kind of heat map, so the more the quotient value deviates from 1.0, the more likely water or ice is detected there.
[0151] As a third step, the processor 120 of the device 200 detects or locates the area including water. The location or detection of the area including water can be performed on a pixel-by-pixel basis or via "feature extraction" (i.e., identification of continuous areas) with a threshold as the final decision. The detected area 140 including water is the output of the evaluation method 280 of the device 200.
[0152] It should be noted here that the water absorption rate is approximately That is, a 1 mm deep water film with a pixel saturation of 255 can only absorb 2.0·10 -3 ·255=0.51.
[0153] In order to fully measure the absorption effects, neighboring pixels are also combined in a pixel window. The larger the pixel window, the more the absorption effects will be amplified and, therefore, the resulting folded image or water heat map will become more blurred. For example, if a puddle consists of only a single pixel or too few pixels, it will not be recognized.
[0154] Furthermore, the difference between simple difference calculations and simple quotient calculations is explained in more detail in the following examples.
[0155] As a starting point, a surface with light and dark areas is chosen, which is covered by a film of water. In the spectral range where water absorption is almost negligible, a measured value of, for example, 200 is obtained for the light areas and a measured value of, for example, 50 is obtained for the dark areas. In the spectral range where water absorbs to a significant extent, a reading of, for example, 0.8·200=160 is obtained for the light areas and a reading of 0.8·50=40 is obtained for the dark areas.
[0156] Now, for the light areas the difference is 40, and for the dark areas the difference is 10, which will lead to the wrong conclusion that there is more water on the light areas.
[0157] However, for both areas the quotient results in a value of 0.8. In the case of the quotient calculation, the dry area corresponds to a value of 1.0.
[0158] For example, pixel-wise normalization of the difference is also possible, e.g. Then y will simply be 1-the quotient. The values i1(x,y) and i2(x,y) represent individual pixel intensity values of the first image and the second image at position x,y.
[0159] Figure 6 An exemplary scenario of an absorption measurement performed by the device 100 is shown. Figure 6 a shows the original surface, ie the picture without the filter, with an area comprising water 610a.
[0160] Figure 6 b shows a combined image or folded image. The combined image is produced by performing a pixel-by-pixel quotient calculation between a first image 143 having a spectral range between 900 nm and 1000 nm and a second image 146 having a spectral range between 800 nm and 900 nm. In order to be able to produce a reduced picture, the pixels are averaged over a 5×5 pixel window.
[0161] In the example, the following bandpass filters are used:
[0162] 850nm CWL, 50nm FWHM bandpass filter and 400-2200nm vertical polarization filter with a contrast ratio of 400:1, and
[0163] 950nm CWL, 50nm FWHM bandpass filter and 400-2200nm vertical polarization filter with a contrast ratio of 400:1.
[0164] The combined image was evaluated using a simple threshold. For example, the threshold decision can be defined using the following function:
[0165] max(round(i(x,y)-0.4),1.0)
[0166] Dry or milky areas in the image appear light or white and have values greater than 0.9. Values less than 0.9 appear dark or black in the image and are identified as water. Clean / wet ice absorbs like water.
[0167] Light may not be able to penetrate milky ice or snow, so milky ice or snow has no effect on absorptivity.
[0168] The area including water 610b is clearly visible in the combined image. Furthermore, the reflection of the trees on the water surface barely affects the absorption measurement. The combined image corrects the measurement to change external influences (such as the environment of underground materials, lighting or changes in reflections in the water body).
[0169] Clean ice is like water.
[0170] This absorption measurement method is configured to cope with unknown light sources. The only requirement is that the light source has sufficient energy in the near infrared (NIR) range (i.e. below 1000nm). In sunlight there is no problem at all, and at night, for example, car headlights (such as halogen lamps) are sufficient.
[0171] The absorption peak of water is around 980 nm, so a first bandpass filter is used in the wavelength range between 900 and 1000 nm.
[0172] In practice, the second band pass filter is freely selectable within the wavelength range between 400 and 900 nm. In an embodiment, the second band pass filter is within the wavelength range between 800 and 900 nm.
[0173] Therefore, in simple terms, two colors are compared. However, the optical bandpass filters 133, 136 used are not in the visible range. The optical sensor 110 used (e.g., a CMOS sensor) does not have a Bayer filter coating and is therefore monochromatic. The three colors red, green, and blue (RGB) are only bandpass filters with wavelength ranges of ~600-700nm, ~500-600nm, and ~400-500nm.
[0174] Figure 7 shows the reflective characteristics of water 760 as detected by device 100. All forms of water exhibit significant specular reflection, which causes polarization effects.
[0175] Figure 7a An unpolarized light ray 710 is shown partially penetrating water 760 and partially reflecting off the water surface.
[0176] Figure 7b A graph showing the reflectance of light rays polarized perpendicularly (TE) 753 and parallel (TM) 756 and reflected off the water surface at different incident angles 740 is shown. The incident angle perpendicular to the water surface has a value of 0°. In addition, the Brewster angle 750 is approximately 53° for water, and also Figure 7b With Brewster angle 750, the reflected ray is 100% horizontally polarized. As long as the incident angle 740 of the reflected ray is close to Brewster angle 750, the reflected ray 730 will be 100% horizontally polarized.
[0177] Figure 7cThe reflectance ratio of vertically and parallelly polarized light reflecting off the water surface at different incident angles 740 is shown. The polarization rate has a maximum value at the Brewster angle 750 (ie, about 53°).
[0178] exist Figure 7a In FIG. 7 , the light ray 730 reflected off the water surface is primarily horizontally polarized, while the penetrating light ray 720 is primarily vertically polarized.
[0179] Only the penetrating light 720 is affected by the absorption of water. The horizontally polarized rays 730 reflected in a specular manner are irrelevant to the absorption measurement or even interfere with the absorption measurement. Figure 1 By using a vertical polarization filter 115 in the optical system, it is possible to reduce or eliminate horizontally polarized rays 730 that are reflected in a specular manner.
[0180] like Figure 7b As shown in , a vertical polarizing filter is still beneficial as long as the incident angle 740 of the reflected ray is close to the Brewster angle 750, because here the reflected ray 730 is 100% horizontally polarized. The further the incident angle is from the Brewster angle, the less effective the polarizing filter is.
[0181] If the image sensor of a mobile tool (eg, mobile tool 300 in FIG. 3 ) wants to look 100 m ahead at a camera height of 1.40 m, the angle of incidence will be approximately -89°.
[0182] In the following, backscatter measurements will be explained.The evaluation 290 of the backscatter measurements is performed by the processor 120 of the device 200 when using the third, fourth and fifth images, which differ from each other in terms of their polarization planes.
[0183] The polarization angle of an image can be expressed as a polarization ellipse. Figure 8 a shows that the polarization ellipse cannot be determined explicitly based on the two polarization planes 810, 820, since there are always three possible solutions 830, 840, 850. With the help of two polarization planes, it is still possible to train a model that achieves the possible results.
[0184] Figure 8 b shows three polarization planes 860, 870, 880 defining a polarization ellipse 890. The three polarization planes 860, 870, 880 are evenly distributed and the angular offset of the polarization planes is insignificant here.
[0185] Figure 8 c and 8d show the Figure 2The processor 120 of the device 200 performs an evaluation 290 or classification of the backscattering characteristics. In the evaluation 290 of the backscattering characteristics, the relationship between the major axis and the minor axis of the polarization ellipse is important. Here, in most cases, two polarization filters (0° and 90°) are sufficient. A third filter allows to improve the accuracy. The tilt of the ellipse depends greatly on the ambient light and is therefore difficult to use.
[0186] As in Figure 7a As explained in , rays reflected off the water surface are predominantly horizontally polarized. Figure 8 c shows the polarization ellipse of liquid water with a large horizontal polarization ratio.
[0187] The ice crystals scatter the light and rotate the polarization ellipse. The result is a slightly shifted, more diffuse polarization. Figure 8 d shows the polarization ellipse of solid water with a uniform polarization component.
[0188] The rotation of the polarization ellipse depends on the illumination. Under uniform illumination, the rotation of the polarization ellipse is also uniform and can be used as a feature.
[0189] Figure 8 f shows that, with uniform illumination, ice crystals scatter the light and rotate the polarization ellipse. At the same time, Figure 8 e shows the polarization ellipse of water without any change in orientation.
[0190] Fig. 9 A scene of an exemplary backscatter measurement with three exemplary representations and an original picture is shown. Fig. 9 d shows the original surface, ie the picture without the filter. Fig. 9 d shows a road surface having a dry area 920 , an area comprising water 910 and two icy surfaces 930 .
[0191] Fig. 9 9a, 9b, 9c show the third, fourth and fifth images respectively, which differ from each other in their polarization planes. Fig. 9 ac is a false color representation. Here, the picture from the polarizing filter is placed on R (+60°), G (0°), B (-60°) to make the effect more vivid. Here, white (R~G~B) is ice, green (R <g>B) is water, while purple (R > G < B) is dry.
[0192] The following polarization filters were used in the examples:
[0193] · A 400 - 2200 nm polarization filter with a contrast ratio of 400:1 at 0° (horizontal).
[0194] · A 400 - 2200 nm polarization filter with a contrast ratio of 400:1 at 60°.
[0195] · A 400 - 2200 nm polarization filter with a contrast ratio of 400:1 at 120°.
[0196] The backscatter measurement is independent of the absorption measurement. Here, the three captured images are normalized on a pixel - by - pixel basis to the average of the three images and compared to each other. In the normalization, the following equation is used: where p 1N represents the normalized pixel of the first image, and p1, p2, and p3 represent the respective picture - element values of the third, fourth, and fifth images.
[0197] In this case, a simple threshold decision would be, for example: if p0 >> p 60 and p0 >> p 120 , then it is water, or if p0 ∼= p 60 ∼= p 120 , then it is milky ice / snow, or if p0 > p 60 ∼= p 120 , then it is clean ice. The values p0, p 60 and p 120 represent the respective picture - element values of the third, fourth, and fifth images.
[0198] In the first scenario, the heat maps of the various methods are simply combined. The backscatter method measures surface scattering and interprets any smooth object as water. The absorption method will ignore this.
[0199] Only when the absorption rate is high and the backscatter measurement determines little or no scattering or a smooth surface is the liquid water at hand.
[0200] In the case of clean ice, the logic is different. When the absorption rate is high (i.e., there is water) and the backscatter measurement shows high scattering, the clean ice is at hand.
[0201] Cloudy ice / snow is difficult to detect by absorption, but is easier to detect by backscatter because it exhibits no absorption but very high scattering.
[0202] The absorption method can be improved by using an algorithm that evaluates neighboring pixels and a trained decision model that takes into account all surroundings. In order to cover different situations, the model is trained here with the help of machine learning. For image segments with a size of 32×32 pixels, the recognition rate reaches 98%, and when cross-validated with field test data, the recognition rate still reaches a very good 85%. When using a neural network that also evaluates the shape, and when training is performed with field test data, more than 95% is achieved again.
[0203] Fig.10 A schematic diagram of an embodiment of an optical sensor 1000 is shown including an image sensor 1020 and including a filter layer 1010. The filter layer 1010 is located over the image sensor 1020. The image sensor is divided into pixels 1030.
[0204] The filter layer is evenly divided into four different filter groups on a pixel-by-pixel basis, namely F1, F2, F3, F4. For example, filter groups F1 and F2 may include bandpass filters, while filter groups F3 and F4 may include polarizing filters.
[0205] The light 1040 will reach the filter layer 1010. The pixels 1030 of the image sensor 1020 will receive the filtered light from the different filter groups F1, F2, F3, F4 of the filter layer 1010. The image produced by the individual sensor is divided into four pictures 1060, 1070, 1080, 1090 by the filter groups F1, F2, F3, F4, and the image is provided to the processor.
[0206] Optical sensor 1000 may replace sensors 110 and 210 in device 200 .
[0207] Although some aspects have been described in the context of devices, it should be understood that the aspects also represent descriptions of corresponding methods, and therefore, blocks or structural components of devices should also be understood as corresponding method steps or as features of method steps. By analogy, aspects that have been described in conjunction with method steps or as method steps also represent descriptions of corresponding blocks or details or features of corresponding devices. Some or all of the method steps can be performed by hardware devices (or performed when using hardware devices). In some embodiments, some or several of the most important method steps can be performed by such devices.
[0208] Fig.11 An embodiment involving three different cameras or photosensitive areas is shown. A first camera 1101, a second camera 1102, and a third camera 1103 are arranged in close proximity to each other to sense the same light shown by Xi. The first camera 1101 has two filter layers 1101a and 1101b. The second camera 1102 has two filter layers 1102a and 1102b, and the third camera 1103 has two filter layers 1103a and 1103b. The filters or filter layers 1101a and 1103a are identical and filter about a first wavelength λ1. The filter layer 1102a or filter 1102a filters about a different wavelength λ2. Furthermore, the filters or filter layers 1101b, 1102b and 1103b are polarization filters, whose elements 1101b and 1102b are identical and filter about a first polarization φ1, while element 1103b filters about a second polarization φ2. The results for each camera and / or photosensitive area 1101, 1102, 1103 are plotted at 1104. Three images have been produced, each image comprising a different combination of polarization (φ1, φ2) and absorption (λ1 and λ2).
[0209] For the evaluation method, requirements and ΔXλ, as presented in brackets on the left side of the box in Equation 1105. In a 4-camera / filter system, the dimensions are obtained directly. Fig.11 In the present 3-camera system shown in , the size is additionally attenuated by another filter, as shown on the left side of the box in equation 1105, because the size and ΔXλ are attenuated by the first absorption filter plotted by λ1(), and / or by the first polarization filter plotted by φ1(). However, since the two filters λ1() and φ1() are known, the expression on the left side of equation 1105 is calculated back to the original response and ΔXλ.
[0210] It should be noted that it does not matter whether the arrangement comprises three dedicated cameras or one chip with 3 areas (pixels) and / or three areas 1102, 1102, 1103 per pixel.
[0211] exist Fig.11 In a preferred embodiment, the following is used:
[0212]
[0213]
[0214] λ1=850nm(Bandpass 50nm FWHM)
[0215] λ2=950nm(Bandpass 50nm FWHM)
[0216] Thus, the preferred device comprises an optical sensor having photosensitive areas (1101, 1102, 1103) for generating a first image (X) of a surface having a first optical bandwidth (λ1). 01 ) and a second image (X ) of the surface having a second optical bandwidth (λ2) 02 ). Moreover, the additional optical sensor is configured to transform the third image (X) having the first polarization filter into 01 ) is provided to the processor, and a fourth image (X 03 ) is provided to the processor. In addition, the processor is configured to determine a difference (X) from the first image and the third image 01 -X 02 ), and calculating the combined image when using the first polarization filter. In addition, the processor is configured to determine a further difference (X) from the third image and the fourth image. 01 -X 03 ), and calculates the polarization difference (ΔXλ) when using the first absorption filter, so that the polarization difference (ΔXλ) is used for evaluating the area including water.
[0217] In the case of the 3-camera / element solution, the first image and the third image referred to for the purposes of this application are the same image. However, the second image and the fourth image are different images of different photosensitive areas 1102 and 1103. Furthermore, it should be noted that the combined image may therefore not necessarily be obtained by simple difference, but by processed difference, possibly in the case of Fig.11 is obtained when using the filter as in the embodiment.
[0218] Depending on the specific implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray disc, CD, ROM, PROM, EPROM, EEPROM or flash memory, hard disk, or any other magnetic or optical storage medium having electronically readable control signals stored thereon, which may interact with a programmable computer system, or may actually interact with a programmable computer system, so as to perform the corresponding method. Thus, the digital storage medium may be computer readable.
[0219] Therefore, some embodiments according to the present invention include a storage medium having electronically readable control signals capable of interacting with a programmable computer system such that any of the methods described herein are performed.
[0220] Generally speaking, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative to perform any method when the computer program product runs on a computer.
[0221] For example, the program code may also be stored on a machine-readable medium.
[0222] Other embodiments comprise the computer program for performing any of the methods described herein, wherein the computer program is stored on a machine readable medium.
[0223] In other words, an embodiment of the inventive method is, therefore, a computer program comprising a program code for performing any of the methods described herein, when the computer program runs on a computer.
[0224] A further embodiment of the method according to the invention is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) on which is recorded the computer program for performing any of the methods described herein.
[0225] Therefore, another embodiment of the method of the invention is a data stream or a sequence of signals representing the computer program for performing any of the methods described herein.The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication link, such as via the Internet.
[0226] Another embodiment comprises a processing device, such as a computer or a programmable logic device, configured to or adapted to perform any of the methods described herein.
[0227] Another embodiment comprises a computer having installed thereon the computer program for performing any of the methods described herein.
[0228] Another embodiment according to the invention includes a device or system configured to transmit a computer program for performing at least one of the methods described herein to a receiver. The transmission may be, for example, electronic or optical. The receiver may be, for example, a computer, a mobile device, a storage device, or the like. The device or system may include, for example, a file server for transmitting the computer program to the receiver.
[0229] In some embodiments, a programmable logic device (e.g., a field programmable gate array, FPGA) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can interact with a microprocessor to perform any method described herein. Generally speaking, in some embodiments, these methods are performed on a part of any hardware device. This can be general hardware (such as a computer processor (CPU)), or hardware dedicated to the method (such as an ASIC).
[0230] The above embodiments are merely illustrative of the principles of the present invention. It should be understood that modifications and variations of the arrangements and details described herein will be apparent to other persons skilled in the art. Therefore, it is intended that the present invention be limited only by the scope of protection of the following claims, rather than by the specific details given by the description and explanation of the embodiments herein.
[0231] literature:
[0232] ALRankin, LHMathies and P.Bellutta (2011), "Daytime water-detectionbased on sky reflections", Robotics and Automation (ICRA 2011IEEE International Conference on. IEEE, 2011, pp. 5329-5336
[0233] CVNguyen, M.Milford, R.Mahony(2017): "3[)tracking of water hazards with polarized stereo cameras", 2017IEEE International Conference on Robotics and Automaton (ICRA) Singapore, May 29th – June 3rd, 2017
[0234] V.Vikari,T.Varpula,and M.Kantanen(2009),"Road-condition recognitionusing 24-GHz automotive radar",IEEE Transactions on IntelligentTransportation Systems,Vol.10,No.4,pp.639-648,2009
[0235] J.Casselgren,S.Rosendahl,M.Sjòdahl,P.Jonsson(2015),"Road conditionanalysis using NIR illumination and compensating for surrounding light",Optics and Lasers in Engineering 77(2016) pp. 175-182
[0236] 一种用于道路状况检测的近红外光电方法 L.Colace,F.Santoni,G.Assanto,非线性光学与光电子实验室,罗马第三大学,瓦萨卡纳瓦莱大道84号,00146罗马,意大利
[0237] 使用偏振立体相机对水患进行三维跟踪
[0238] Chuong V.Nguyen1,Michael Milford2 and Robert Mahony
[0239] 一种用于驾驶员辅助的车道检测视觉模块
[0240] A.L.Rankin,L.H.Matthies和P.Bellutta(2011),“基于天空反射的白天水检测”,机器人与自动化(ICRA 2011 IEEE国际会议)。IEEE,2011,第5329 - 5336页
[0241] C.V.Nguyen,M.Milford,R.Mahony(2017):“使用偏振立体相机对水患进行三维跟踪”,2017年IEEE国际机器人与自动化会议(ICRA),新加坡,5月29日–6月3日,2017
[0242] V. Vikari, T. Varpula, and M. Kantanen (2009), "Road-condition recognition using 24-GHz automotive radar", IEEE Transactions on Intelligent Transportation Systems, Vol. 10, No. 4, pp. 639-648, 2009
[0243] J. Casselgren, S. Rosendahl, M. Sjòdahl, P. Jonsson (2015), "Road condition analysis using NIR illumination and compensating for surrounding light", Optics and Lasers in Engineering 77 (2016) pp. 175-182
[0244] A near-infrared optoelectronic approach to detection of road conditions L. Colace, F. Santoni, G. Assanto, NooEL - Nonlinear Optics and OptoElectronics Lab, University "Roma Tre", Via della Vasca Navale 84, 00146 Rome, Italy
[0245] 3D tracking of water hazards with polarized stereo cameras
[0246] Chuong V. Nguyen1, Michael Milford2 and Robert Mahony
[0247] A Lane Detection Vision Module for Driver Assistance
[0248] Author: Kristijan; Williams, Brian; Kolski, Sascha; Siegwart, Roland
[0249] 基于颜色变化的日间水检测
[0250] Arturo Rankin and Larry Matthies
[0251] 基于天空反射的日间水检测
[0252] Arturo L.Rankin,Larry H.Matthies和Paolo Bellutta
[0253] 用于自主越野导航的水害检测
[0254] Larry Matthies*,Paolo Bellutta,Mike McHenry
[0255] 通过分析轮胎与道路摩擦超声噪声检测道路上的冰
[0256] D.Gailius,S.
[0257] 使用红外测温法检测路面结冰情况
[0258] Mats Riehm, Gustavsson, Bogren,Per-Erik Jansson
[0259] 用于高级驾驶辅助系统(ADAS)的湿地区域和水坑检测
[0260] (ADAS)使用立体相机
[0261] Jisu Kim,Jeonghyun Baek,Hyukdoo Choi,and Euntai Kim*
[0262] 冰的多光谱成像
[0263] 丹尼斯·格雷戈里斯、西蒙·于和弗兰克·泰蒂
[0264] 基于红外光学成像技术的近场冰检测
[0265] 哈齐姆·阿卜杜勒 - 莫阿蒂、乔纳森·莫里斯、曾佑生、马丁·韦斯利·科里尔、维克多·加拉斯·亚尼
[0266] 检测道路结冰形成的新系统
[0267] 阿梅代奥·特罗亚诺、埃罗斯·帕塞罗、IEEE会员和卢卡·梅西恩
[0268] 用于通过高光谱成像进行冰检测的带通滤波器最佳组合的选择
[0269] 中内茂树、西野健和山下拓也
[0270] 用于光学检测危险道路状况的传感器
[0271] 阿曼多·皮卡迪和洛伦佐·科拉切
[0272] 用于
[0273] 自主越野导航的基于偏振的水害检测
[0274] 谢斌、潘华东、向志宇、刘吉林
[0275] 使用近红外照明进行道路状况分析并补偿环境光
[0276] 约翰·卡塞尔格伦、萨拉·罗森达尔、米凯尔 , 帕特里克·琼森
[0277] 河流场景的自监督分割
[0278] 苏普里特·阿查尔、巴拉特·桑卡兰、斯蒂芬·努斯克、塞巴斯蒂安·舍勒和桑吉夫·辛格。< / g>
Claims
1. A device (100, 200) for identifying water (520, 610a, 610b, 760, 910) on a surface (150), comprising: An optical sensor (110, 1000) for generating a first image (143) of the surface having a first optical bandwidth, and a second image (146) of the surface having a second optical bandwidth, wherein the water has a first absorptivity in the first optical bandwidth and the water has a second absorptivity in the second optical bandwidth that is higher than the first absorptivity; a processor (120) for combining the first image and the second image to produce a combined image (130, 530) in which the surface is reduced or eliminated compared to the water, and for detecting the water in the combined image; as well as an additional optical sensor configured to provide a third image to the processor using a first polarization filter having a first polarization angle and to provide a fourth image to the processor using a second polarization filter having a second polarization angle, wherein the two polarization filters differ from each other with respect to their polarization angles, and wherein the processor is configured to detect the aggregation state of the water when using the third image and the fourth image in the combined image, wherein the optical sensor (1000) and the additional optical sensor are configured in an individual sensor (1020) comprising a filter layer (1010), wherein the filter layer (1010) is configured to divide pixels of an image generated by the individual sensor into four groups in a row-by-row, column-by-column or checkerboard manner, wherein a first optical bandpass filter (F1) having a first bandwidth is arranged above a first group of pixels such that said pixels of said first group generate said first image having said first optical bandwidth, wherein a second optical bandpass filter (F2) having a second bandwidth is arranged above the pixels of the second group such that said pixels of said second group generate said second image having said second optical bandwidth, wherein the first polarization filter (F3) is arranged above pixels of a third group so that the pixels of the third group generate the third image having the first polarization angle, and The second polarization filter (F4) is arranged above the pixels of the fourth group, so that the pixels of the fourth group generate the fourth image with the second polarization angle.
2. The apparatus of claim 1, wherein the first optical bandwidth comprises a bandwidth selected from a spectral range between 400 nm and 900 nm; and the second optical bandwidth comprises a bandwidth selected from a spectral range between 900 nm and 1200 nm.
3. The apparatus of claim 1, wherein the first optical bandwidth has a value between 820 nm and 870 nm at half width; and the second optical bandwidth has a value between 920 nm and 970 nm at the half width.
4. The device of claim 1, wherein the optical sensor comprises a silicon-based image sensor.
5. The device of claim 1, wherein the optical sensor comprises a CMOS sensor.
6. The device of claim 1, wherein the processor is configured to use a pixel-by-pixel quotient calculation, a pixel-by-pixel difference calculation, a pixel-by-pixel normalized difference calculation, or a difference between 1 and a quotient calculation when combining the first image and the second image.
7. The apparatus of claim 1, wherein the processor is configured to generate a folded image, Wherein the processor is configured to scan the combined image having pixels on a pixel-by-pixel basis or on a block-by-block basis by using a pixel window to calculate a weighted average of the values of the pixels of the pixel window, and generate a folded image based on the weighted average, wherein a position of the weighted average in the folded image matches a position of the pixel window in the combined image.
8. A device as claimed in claim 1, wherein the processor is configured to compare the values of individual pixels of the combined image or a folded image derived from the combined image with a threshold value, wherein pixels having values having a certain relationship with the threshold value are detected as water, or feature detection is performed to detect water.
9. The apparatus of claim 1, wherein the polarization angles of the polarization filters are arranged to be offset by 90° from each other.
10. The apparatus of claim 1, wherein the processor is configured to detecting an area comprising water in the combined image or in a folded image derived from the combined image, evaluating pixels from the third image and the fourth image in the region comprising water, and The state of the water aggregation is detected, including whether the region of water has liquid water or ice.
11. The apparatus of claim 10, wherein the processor is configured to generating normalized intensity values of the third image and the fourth image when evaluating the pixels of the third image and the fourth image, in generating the normalized intensity value, using a comparison of the value of the pixel with the sum of the values of the pixel of the third image and the fourth image, Detecting the aggregation state of the water, the processor is configured to When the normalized intensity values of the third image and the fourth image are within a range of + / - 10% including the mean of the normalized intensity values of the third image and the fourth image, the aggregation state of the water is detected to be ice, or When a normalized intensity value of the normalized intensity values from the third image and the fourth image is greater than 1.5 times the average of the normalized intensity values of the third image and the fourth image, the aggregation state of the water is detected to be liquid water.
12. The device of claim 1, comprising a temperature sensor (230) configured to provide an ambient temperature value to the processor, by means of which the processor can assess the state of accumulation of the water.
13. The apparatus of claim 1, wherein the processor is configured to sensing a set of state values including information from the first image and the second image and providing a variable threshold value based on the set when using an artificial intelligence element or a neural network, and Water in the combined image or a state of aggregation of the water in the combined image is detected when the variable threshold is used.
14. The apparatus of claim 1, wherein the processor is configured to sensing a set of state values comprising information from the third image and from the fourth image, and providing a variable threshold value based on the set when using an artificial intelligence element or a neural network, and Water in the combined image or a state of aggregation of the water in the combined image is detected when the variable threshold is used.
15. A mobile tool (300) comprising a device as claimed in claim 1 and an interface (310), wherein the interface (310) is configured to warn a driver (320) of the mobile tool (300) and / or influence control of the mobile tool (300) if the device detects a solid state of the water.
16. A method of distinguishing between liquid and solid aggregate states of water in an area comprising water, the method comprising: obtaining a first image of a surface having a first optical bandwidth, obtaining a second image of the surface having a second optical bandwidth, the water having a first absorptivity in the first optical bandwidth, the water having a second absorptivity in the second optical bandwidth that is higher than the first absorptivity, combining the first image and the second image to obtain a combined image in which the surface is reduced or eliminated compared to the water, detecting said water in said combined image, obtaining a third image using a first polarization filter having a first polarization angle, and obtaining a fourth image using a second polarization filter having a second polarization angle, wherein the two polarization filters differ from each other with respect to their polarization angles, evaluating pixels of the third image and the fourth image in areas comprising water, and detecting the aggregation state of the water based on the evaluation of the pixels, The pixels in the area are divided into four groups in a row-by-row, column-by-column or chessboard manner, wherein a first optical bandpass filter (F1) having a first bandwidth is arranged above a first group of pixels such that said pixels of said first group generate said first image having said first optical bandwidth, wherein a second optical bandpass filter (F2) having a second bandwidth is arranged above the pixels of the second group such that said pixels of said second group generate said second image having said second optical bandwidth, wherein the first polarization filter (F3) is arranged above pixels of a third group so that the pixels of the third group generate the third image having the first polarization angle, and The second polarization filter (F4) is arranged above the pixels of the fourth group, so that the pixels of the fourth group generate the fourth image with the second polarization angle.
17. A computer program product comprising a computer program for executing the method according to claim 16 when the computer program is run on a computer.
