Rainfall measurement method and apparatus, and vehicle

Abstract

A rainfall measurement method and apparatus, and a vehicle. The method comprises: acquiring multi-source information, the multi-source information at least comprising: a plurality of image frames captured by at least one camera apparatus of a vehicle, and rainfall component information, wherein the rainfall component information indicates a gear position of a windscreen wiper of the vehicle and / or rainfall measured by a rainfall sensor of the vehicle; on the basis of the plurality of image frames, determining camera failure information, wherein the camera failure information indicates the degree of impact of the rainfall on the clarity of images captured by each of the at least one camera apparatus; and on the basis of at least the camera failure information and the rainfall component information, determining a rainfall grade. The technical solution can be applied to the field of intelligent vehicles such as electric vehicles and new energy vehicles, and facilitates improvement to the rainfall intensity measurement accuracy of vehicles, thereby providing a reference for whether the vehicles can enable an intelligent driving function.

Description

Rainfall detection method, device and vehicle

[0001] This application claims priority to the Chinese patent application No. 202411292020.0, filed on September 13, 2024, and entitled "Rainfall detection method, device and vehicle", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of intelligent vehicles, and more particularly, to a rainfall detection method, device and vehicle. BACKGROUND

[0003] Rainy days are a very common driving scenario, and rainwater can affect the driver's vision, and the vehicle is easy to slip when driving on the ground wet by rainwater, resulting in traffic accidents in rainy days being about 5 times higher than in non-rainy days. Rainy days also pose challenges to the safety and experience of intelligent driving. Rainwater not only causes the vehicle to slip, but also causes the perception system of the vehicle to decrease in perception ability, thereby causing accidents to occur.

[0004] In view of this, in order to facilitate the intelligent driving system of the vehicle to make accurate decisions and planning, a scheme capable of accurately detecting rainfall is urgently needed to be developed. SUMMARY

[0005] The present application provides a rainfall detection method, device and vehicle, which helps to improve the accuracy of rainfall detection, thereby improving the safety of the vehicle driving in rainfall.

[0006] In a first aspect, a rainfall detection method is provided, which can be executed by a vehicle, for example, can be executed by a computing platform of the vehicle, or can also be executed by a chip or circuit for the vehicle.

[0007] The method comprises: acquiring multi-source information, the multi-source information at least comprising: a plurality of images collected by at least one camera device of the vehicle, and rainfall component information, the rainfall component information indicating a gear position of a rain wiper of the vehicle and / or a rainfall detected by a rainfall sensor of the vehicle; determining camera failure information according to the plurality of images, the camera failure information indicating an influence degree of the rainfall on the image definition collected by each camera device in the at least one camera device; and determining a rainfall grade according to at least the camera failure information and the rainfall component information.

[0008] In the above technical solution, the grade of the rainfall is determined according to the multi-source information including the images collected by the camera device and the information of the rainfall component, which helps to improve the detection accuracy of the rainfall grade and improve the applicability of the rainfall detection method in different scenarios.

[0009] In some implementations of the first aspect, the camera failure information includes an image quality score, a water stain score, and a water stain segmentation ratio corresponding to each image captured by each camera, where the image quality score indicates a quality of the image, the water stain score indicates a probability that the quality of the image is affected by the water stain, and the water stain segmentation ratio indicates a proportion of pixels associated with the water stain in the image; and determining the rain intensity level includes determining the rain intensity level based on the image quality score, the water stain score, the water stain segmentation ratio, and the rain component information.

[0010] In some implementations, the image quality score is affected by multiple factors including the rain intensity, such as the image quality score being affected by factors such as physical obstruction and lens flare in addition to the water stain. Therefore, in combination with the water stain score and the water stain segmentation ratio, the influence of the rain intensity on the image clarity can be more accurately determined, and the accuracy of the detection result of the rain intensity level can be improved.

[0011] In some implementations, the multiple images from different cameras are captured in the same time period.

[0012] In the above technical solution, when the rain intensity is less than a certain threshold, the water stain score and the water stain segmentation ratio change with the increase and decrease of the rain intensity, and therefore, in combination with the water stain score and the water stain segmentation ratio, the accuracy of the rain intensity detection result can be improved.

[0013] In some implementations of the first aspect, the multiple images include a first set of images captured by a first camera, and the camera failure information includes an image quality score of each image in the first set of images, where the image quality score indicates a quality of each image; and determining the rain intensity level based on at least the camera failure information and the rain component information includes determining at least one feature based on the image quality score of each image, where the at least one feature includes a score mean, a standard deviation, a first frequency feature, and a second frequency feature, the score mean is an average of the image quality scores of the first set of images, the standard deviation is a standard deviation of the image quality scores of the first set of images, the first frequency feature is a frequency and / or a number of consecutive frames at which the image quality score is greater than or equal to a first score threshold, and the second frequency feature is a frequency and / or a number of consecutive frames at which the image quality score is less than or equal to a second score threshold; and determining the rain intensity level based on the at least one feature and the rain component information.

[0014] Since the water stain score and the water stain segmentation ratio do not change much with the increase and decrease of the rain intensity when the rain intensity is greater than a certain threshold, and the image quality score is more likely to be affected by the rain intensity when the rain intensity is greater, in the above technical solution, based on the multi-dimensional feature extraction of the image quality score, the accuracy of the detection result of the rain intensity level can be improved with a simpler processing logic.

[0015] In some implementations, when the gear position of the wiper is less than or equal to the gear threshold, the rain intensity level is determined according to the image quality score, the water stain score, the water stain segmentation ratio, and the rain intensity component information; and when the gear position of the wiper is greater than the gear threshold, the rain intensity level is determined according to the image quality score and the rain intensity component information, and more specifically, the rain intensity level is determined according to at least one of the foregoing implementations and the rain intensity component information.

[0016] In combination with the first aspect, in some implementations of the first aspect, the field of view angle coverage area of the first camera device includes a windshield area swept by the wiper, and the first frequency feature and / or the second frequency feature are associated with a unit of the wiper.

[0017] In the case of light rain, there is no or less water stain in the image(s) captured by the camera device within a time period after the windshield is swept by the wiper, so the image quality score corresponding to the image(s) is relatively high. As the water stain accumulates, the image quality score continues to decrease until the next time the windshield is swept by the wiper. Therefore, the image quality score fluctuates periodically with the wiper swing frequency. In the case of heavy rain (heavy rain or storm), the number of frames with a high image quality score is small after the wiper is swept, or the image quality score does not fluctuate. Therefore, according to the association between the first frequency feature and the second frequency feature and the swing speed (i.e., the gear position) of the wiper, the rain intensity is detected, which facilitates distinguishing between the scenarios of heavy rain (such as heavy rain or storm) and light rain (such as light rain or moderate rain), thereby improving the accuracy of the rain intensity detection result.

[0018] In combination with the first aspect, in some implementations of the first aspect, each image includes a first pixel area and a second pixel area, the first pixel area includes pixels corresponding to all or part of the windshield area swept by the wiper, and the second pixel area includes pixels corresponding to an area other than the windshield area swept by the wiper; and the area quality score of each image is greater than or equal to a third score threshold; wherein the area quality score indicates the image quality of the first pixel area.

[0019] Since the wiper blockage can cause the image quality score to abnormally decrease, the above technical solution can exclude the image including the pixels corresponding to the wiper, i.e., the image not including the pixels corresponding to the wiper is used for rain intensity detection, which helps to further improve the accuracy of the rain intensity detection result.

[0020] In combination with the first aspect, in some implementations of the first aspect, the plurality of images include a second group of images captured by a second camera device, and the second camera device is configured to capture images in front of the vehicle. The method further includes: determining a visual rain intensity level according to the second group of images; and determining the rain intensity level includes: determining the rain intensity level according to the camera failure information, the rain intensity component information, and the visual rain intensity level.

[0021] In the technical solution, the visual rain intensity level is determined to determine the final rain intensity detection result, which helps to improve the accuracy of the rain intensity detection result. In addition, when the rain intensity is detected based on the images collected by the multiple cameras, the rain intensity level can be detected even if the data source of one of the multiple cameras is lost, and then the rain intensity level is used for control, which helps to ensure the driving safety of the vehicle in the case of heavy rain.

[0022] In combination with the first aspect, in some implementations of the first aspect, the second camera is a wide-angle camera.

[0023] In the technical solution, the images collected by the wide-angle camera can obtain more information outside the vehicle, so that the detection result of the visual rain intensity level is more accurate, thereby improving the accuracy of the rain intensity detection result.

[0024] In combination with the first aspect, in some implementations of the first aspect, the multiple images include a third set of images collected by a third camera, the third camera is configured to obtain images in front of the vehicle, each image in the third set of images includes pixels corresponding to the road surface, and the method further includes: determining the road surface wetness degree according to the third set of images; and determining the rain intensity level, including: determining the rain intensity level according to the camera failure information, the rain intensity component information, and the road surface wetness degree.

[0025] In the technical solution, the visual rain intensity level is determined to determine the final rain intensity detection result, which helps to improve the accuracy of the rain intensity detection result. In addition, when the rain intensity is detected based on the images collected by the multiple cameras, the rain intensity level can be detected even if the data source of one of the multiple cameras is lost.

[0026] In combination with the first aspect, in some implementations of the first aspect, the third camera is a long-focus camera.

[0027] In the technical solution, the images collected by the long-focus camera can obtain more road surface information, i.e., more road surface related pixels in the images, so that the detection result of the road surface wetness degree is more accurate, thereby improving the accuracy of the rain intensity detection result.

[0028] In a second aspect, a rain intensity detection apparatus is provided, comprising an obtaining unit and a processing unit, wherein the obtaining unit is configured to obtain multi-source information, the multi-source information comprising at least: a plurality of images captured by at least one camera of a vehicle, and rain component information, the rain component information indicating a wiper position of a wiper of the vehicle and / or a rain intensity detected by a rain sensor of the vehicle; and the processing unit is configured to determine camera failure information according to the plurality of images, the camera failure information indicating an influence degree of rain on image clarity of each of the images captured by the at least one camera; and determine a rain intensity level according to at least the camera failure information and the rain component information.

[0029] With reference to the second aspect, in some implementations of the second aspect, the camera failure information comprises an image quality score, a water stain score, and a water stain segmentation ratio corresponding to each of the images captured by the at least one camera, wherein the image quality score indicates a quality of the image, the water stain score indicates a probability that the quality of the image is affected by water stain, and the water stain segmentation ratio indicates a proportion of pixels associated with water stain in the image relative to all pixels; and the processing unit is configured to determine the rain intensity level according to the image quality score, the water stain score, the water stain segmentation ratio, and the rain component information.

[0030] With reference to the second aspect, in some implementations of the second aspect, the plurality of images comprises a first set of images captured by a first camera, and the camera failure information comprises an image quality score of each of the first set of images, the image quality score indicating a quality of each of the first set of images; and the processing unit is configured to determine at least one feature according to the image quality score of each of the first set of images, the at least one feature comprising: a score mean, a standard deviation, a first frequency feature, and a second frequency feature; wherein the score mean is an average of the image quality scores corresponding to the first set of images, the standard deviation is a standard deviation of the image quality scores corresponding to the first set of images, the first frequency feature is a frequency and / or a number of consecutive frames that the image quality score is greater than or equal to a first score threshold, and the second frequency feature is a frequency and / or a number of consecutive frames that the image quality score is less than or equal to a second score threshold; and determine the rain intensity level according to the at least one feature and the rain component information.

[0031] With reference to the second aspect, in some implementations of the second aspect, a field of view angle of the first camera covers a windshield area swept by the wiper, and the first frequency feature and / or the second frequency feature is associated with the wiper position.

[0032] With reference to the second aspect, in some implementations of the second aspect, each frame of image includes a first pixel region and a second pixel region, the first pixel region includes pixels corresponding to all or part of the windshield region wiped by the wiper, and the second pixel region includes pixels corresponding to a region other than the windshield region wiped by the wiper; and the region quality score of each frame of image is greater than or equal to the third score threshold; wherein the region quality score indicates the image quality of the first pixel region.

[0033] With reference to the second aspect, in some implementations of the second aspect, the plurality of frames of image includes a second set of images captured by a second camera device, the second camera device is configured to obtain images in front of the vehicle, and the processing unit is further configured to: determine a visual rain amount level based on the second set of images; and determine the rain amount level based on the camera failure information, the rain amount component information, and the visual rain amount level.

[0034] With reference to the second aspect, in some implementations of the second aspect, the second camera device is a wide-angle camera device.

[0035] With reference to the second aspect, in some implementations of the second aspect, the plurality of frames of image includes a third set of images captured by a third camera device, the third camera device is configured to obtain images in front of the vehicle, and each frame of image in the third set of images includes pixels corresponding to a road surface, and the processing unit is further configured to: determine a road surface wetness level based on the third set of images; and determine the rain amount level based on the camera failure information, the rain amount component information, and the road surface wetness level.

[0036] With reference to the second aspect, in some implementations of the second aspect, the third camera device is a long-focus camera device.

[0037] In a third aspect, a rain amount detection device is provided, and the device includes a processor configured to execute a computer program stored in a memory, so that the device performs the method in any possible implementation manner of the first aspect.

[0038] With reference to the third aspect, in some implementations of the third aspect, the device further includes the memory.

[0039] In a fourth aspect, a computer program product is provided, and the computer program product includes computer program code configured to cause a computer or a processor to perform the method in any possible implementation manner of the first aspect when the computer program code is run on the computer or the processor.

[0040] It should be noted that the computer program code can be stored in whole or in part on a storage medium, and the storage medium can be packaged together with the processor or packaged separately from the processor.

[0041] In a fifth aspect, a computer readable storage medium is provided, and the computer readable medium stores instructions which, when executed by a processor, cause the processor to implement the method in any possible implementation of the first aspect.

[0042] In a sixth aspect, a chip is provided, and the chip comprises circuitry configured to implement the method in any possible implementation of the first aspect.

[0043] In a seventh aspect, a vehicle is provided, and the vehicle comprises the apparatus in any possible implementation of the second aspect or the third aspect, or the vehicle comprises the computer readable storage in any possible implementation of the fifth aspect, or the vehicle comprises the chip in any possible implementation of the sixth aspect, or the vehicle is loaded with the computer program code in any possible implementation of the fourth aspect.

[0044] In connection with the seventh aspect, in some implementations of the seventh aspect, the vehicle is a vehicle in a broad sense, for example, can be a vehicle (such as a commercial vehicle, a passenger vehicle, a motorcycle, a flying vehicle, a train, etc.), an industrial vehicle (such as a forklift, a trailer, a tractor, etc.), an engineering vehicle (such as an excavator, a bulldozer, a crane, etc.), an agricultural device (such as a mower, a harvester, etc.), a recreational device, a toy vehicle, etc. In actual implementation, the vehicle can also be a road vehicle, a water vehicle, an air vehicle, an industrial device, an agricultural device, or an entertainment device, or other intelligent driving devices.

[0045] The beneficial effects not described in the second aspect to the seventh aspect can be referred to the description in the first aspect, and will not be described here again. BRIEF DESCRIPTION OF DRAWINGS

[0046] FIG. 1 is a functional schematic block diagram of a vehicle provided by an embodiment of the present application;

[0047] FIG. 2 is a schematic block diagram of a rain system architecture provided by an embodiment of the present application;

[0048] FIG. 3 is a schematic flow chart of a rain detection method provided by an embodiment of the present application;

[0049] FIG. 4 is a schematic diagram of images corresponding to different visual rain grades provided by an embodiment of the present application;

[0050] FIG. 5 is a schematic diagram of a water accumulation area segmentation result provided by an embodiment of the present application;

[0051] FIG. 6 is a schematic diagram of an image captured by a camera device and a result of area division on the image provided by an embodiment of the present application;

[0052] FIG. 7 is a schematic diagram of a result obtained by counting and accumulating according to a rain component provided by an embodiment of the present application;

[0053] FIG. 8 is another schematic diagram of a rain detection system according to an embodiment of the present application;

[0054] FIG. 9 is a schematic diagram of three consecutive images captured by a front-facing camera according to an embodiment of the present application;

[0055] FIG. 10 is another schematic flowchart of a rain detection method according to an embodiment of the present application;

[0056] FIG. 11 is a schematic block diagram of a rain detection device according to an embodiment of the present application;

[0057] FIG. 12 is another schematic block diagram of a rain detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0059] FIG. 1 is a functional block diagram of a vehicle according to an embodiment of the present application. As shown in FIG. 1, the vehicle 100 can include a perception system 120 and a computing platform 150, wherein the perception system 120 can include several sensors for sensing information of the environment around the vehicle 100. For example, the perception system 120 can include a positioning system, which can be a global navigation satellite system (GNSS, such as a global positioning system (GPS), a Beidou system, etc.). For another example, the perception system 120 can also include one or more of an inertial measurement unit (IMU), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera.

[0060] Exemplarily, the perception system 120 can include one or more cameras for capturing images outside the vehicle, specifically, the cameras can include one or more of a front-view camera, a rear-view camera, an all-around-view camera, and a side-view camera. The front-view camera can be installed at an inner rearview mirror inside a front windshield, the front-view camera can include a wide-angle camera and a long-focus camera. The field of view (FOV) of the wide-angle camera can be 130°, and the detection range can be 10-50 meters; the FOV of the long-focus camera can be 30°, and the detection range can be 50-200 meters. The rear-view camera can be installed at a rear trunk. The side-view camera can be installed below the rearview mirror; the all-around-view camera includes four cameras installed around the vehicle, and the images captured by the four cameras can be spliced to obtain a panoramic image around the vehicle. In some implementations, the all-around-view camera can coincide with the front-view camera, the rear-view camera, and the side-view camera, for example, the camera of the all-around-view camera arranged at the side of the vehicle can be the side-view camera, the camera of the all-around-view camera arranged at the front of the vehicle can be the front-view camera, and the camera of the all-around-view camera arranged at the rear of the vehicle can be the rear-view camera. Alternatively, the all-around-view camera can also be different from the front-view camera, the rear-view camera, and the side-view camera.

[0061] Some or all of the functionality of the vehicle 100 can be controlled by the computing platform 150. The computing platform 150 can include processors 151-15n, which are circuits that have the capability to process signals. In one implementation, the processors can be circuits that have the capability to read and execute instructions, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a type of microprocessor), a digital signal processor (DSP), or the like. In another implementation, the processors can be circuits that implement functionality through fixed or reconfigurable logic, such as an application-specific integrated circuit (ASIC) or a programmable logic device (PLD) such as a field programmable gate array (FPGA). In reconfigurable hardware circuits, the processor loads configuration files to implement the configuration of the hardware circuit, which can be understood as the processor loading instructions to implement the relevant functionality. Additionally, the processors can be hardware circuits designed for artificial intelligence, which can be understood as a type of ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), or the like. Additionally, the computing platform 150 can include a memory that stores instructions that can be called by some or all of the processors 151-15n to implement functionality.

[0062] The operation of the intelligent driving system can be controlled by the computing platform 150, which can include an advanced driving assistant system (ADAS) and an autonomous driving system (ADS). The intelligent driving system uses various sensors on the vehicle (including but not limited to laser radar, millimeter wave radar, camera, ultrasonic sensor, global positioning system, inertial measurement unit) to obtain information from the surroundings of the vehicle, and analyzes and processes the obtained information to realize functions such as obstacle perception, target recognition, vehicle positioning, path planning, driver monitoring / alerting, etc., thereby improving the safety, automation level and comfort of vehicle driving.

[0063] At different levels of autonomous driving (or intelligent driving, L0-L5 in total), the intelligent driving system can realize different levels of autonomous driving assistance based on artificial intelligence algorithms and information obtained by multiple sensors. The above-mentioned levels of autonomous driving are based on the classification standard of the Society of Automotive Engineers (SAE). Among them, L0 is non-automation; L1 is driving assistance; L2 is partial automation; L3 is conditional automation; L4 is high automation; and L5 is full automation. The tasks of monitoring the road conditions and responding are completed by the driver and the system together at L1 to L3 levels, and the driver needs to take over the dynamic driving task. At L4 and L5 levels, the driver can completely change into a passenger role. Currently, the functions that the intelligent driving system can realize mainly include but are not limited to adaptive cruise assistance, automatic emergency braking, automatic parking, blind spot monitoring, front intersection traffic warning / braking, rear intersection traffic warning / braking, front vehicle collision warning, lane departure warning, lane keeping assistance, rear vehicle collision warning, traffic sign recognition, traffic congestion assistance, highway assistance, etc. It should be understood that the above-mentioned various functions can have specific modes at different levels of autonomous driving (L0-L5), and the higher the level of autonomous driving, the more intelligent the corresponding mode.

[0064] FIG. 2 shows a schematic block diagram of a rain detection system architecture according to an embodiment of the present application. The system includes a perception module 210, a perception information processing module 220, a rain component detection module 230, and a rain detection module 240. In some implementations, the system further includes a control module 250. Specifically:

[0065] The perception module 210 can include one or more cameras in the perception system 120 shown in FIG. 1, for collecting images that can represent whether the weather of the environment in which the vehicle is currently located is rainy and the amount of rain. The images collected by the perception module 210 can be sent to the perception information processing module 220.

[0066] The perception information processing module 220 can include one or more processors in the computing platform 150 shown in FIG. 1, for performing preliminary processing on the images. Specifically, the perception information processing module 220 includes a camera failure detection module 221, a visual rain amount detection module 222, and a road surface wetness level detection module 223. More specifically, the camera failure detection module 221 can determine an image quality score, a water stain score, and a water stain segmentation ratio based on images collected by a camera disposed outside the vehicle. The image quality score can indicate whether the image quality is reduced due to semi-transparent obstruction, water mist, glare, etc. For example, the lower the image quality score, the worse the image quality and the worse the image clarity. The water stain score indicates the probability that the image quality is affected by water stains. The higher the water stain score, the greater the probability that the image quality is reduced due to water stains. It should be noted that the water stains can be attached to the lens surface of the camera, or when the camera is a front-view camera mounted inside the windshield, the water stains can also be attached to the windshield. The water stain segmentation ratio indicates the proportion of the area of the lens of the camera that has water stains, or the proportion of the area of the windshield within the FOV of the camera that has water stains. The visual rain amount detection module 222 is configured to determine the visual rain amount based on features of images collected by an in-vehicle camera. The road surface wetness level detection module 223 is configured to determine the wetness level of the road based on features of images collected by an in-vehicle camera. Further, the perception information processing module 220 can send the processing results obtained by each module to the rain amount detection module 240.

[0067] The rain amount component detection module 230 can include at least one rain amount related sensor, for example, the rain amount component detection module 230 can include a rain amount sensor 231 and a wiper speed detection module 232. The rain amount sensor 231 can be an optical sensor or other sensor capable of detecting the amount of rain. The wiper speed detection module 232 is configured to determine the swing speed (or gear position) of the wiper. The rain amount component detection module 230 can send the results detected by each module to the rain amount detection module 240.

[0068] The rain detection module 240 determines a final detection result of the rain level according to one or more processing results from the perception information processing module 220 and / or at least one detection result from the rain component detection module 230. In actual implementation, the rain detection module 240 can also determine the final detection result of the rain level in combination with weather information from a third-party weather software, where the weather information is a forecast of the weather condition of the region where the vehicle is located at the current time and in the future for a period of time. In some implementations, the rain detection module 240 can send the detection result of the rain level to the control module 250.

[0069] The control module 250 can include one or more processors in the computing platform 150 shown in FIG. 1, for controlling the enabling and disabling of the intelligent driving function of the vehicle according to the detection result of the rain detection module 240. For example, when the detection result sent by the rain detection module 240 indicates that the rain level is greater than or equal to a preset level, the control module 250 can turn off the intelligent driving function related to the driving of the vehicle.

[0070] It should be understood that the above modules are only an example, and in actual application, the above modules can be added or deleted as needed. For example, in the system architecture shown in FIG. 2, the rain detection module 240 and the control module 250 are combined into one module; for another example, the rain detection module 240 and the perception information processing module 220 are combined into one module.

[0071] The above describes the system related to the embodiments of the present application in combination with FIG. 2, and the following describes in detail the rain detection method based on the system shown in FIG. 2.

[0072] FIG. 3 shows an exemplary flowchart of a sensor detection method provided by the embodiments of the present application, the method 300 can be executed by the system shown in FIG. 2, the method 300 includes S301 to S304, where S301 and S302 can be executed by the perception information processing module 220, S303 can be executed by the rain component detection module 230, and S304 can be executed by the rain detection module 240. Specifically, the method 300 includes:

[0073] S301, acquiring an image collected by at least one camera device.

[0074] For example, the at least one camera device can include one or more of a surround-view camera device, a front-view camera device, a rear-view camera device, or a side-view camera device.

[0075] S302, determining at least one of the following perception information processing results according to the image: a visual rain level, a road surface wetness level, and camera failure information.

[0076] In some implementations, the images captured by the front-facing camera (e.g., the front-facing wide-angle camera) in time period 1 are input into the visual rainfall model to obtain a visual rainfall level. For example, a set of images shown in (a) to (d) in FIG. 4 can be labeled as no rain, light rain, moderate rain, and heavy rain, respectively, and then input into the visual rainfall model to train the visual rainfall model. In this way, when the images captured by the front-facing camera are input into the visual rainfall model, the visual rainfall model can determine the current visual rainfall level as one of no rain, light rain, moderate rain, and heavy rain according to the images. The visual rainfall model can be regarded as an example of the aforementioned visual rainfall detection module 222.

[0077] In some implementations, the images captured by the front-facing camera (e.g., the front-facing wide-angle camera) in time period 1 are input into the visual rainfall model to obtain a visual rainfall level. For example, a set of images shown in (a) to (d) in FIG. 4 can be labeled as no rain, light rain, moderate rain, and heavy rain, respectively, and then input into the visual rainfall model to train the visual rainfall model. In this way, when the images captured by the front-facing camera are input into the visual rainfall model, the visual rainfall model can determine the current visual rainfall level as one of no rain, light rain, moderate rain, and heavy rain according to the images. The visual rainfall model can be regarded as an example of the aforementioned visual rainfall detection module 222.

[0078] For example, taking the pixel of the wet area (i.e., the road surface) in the vehicle coordinate system as Z = 0 plane, according to the pinhole imaging principle, the coordinate values (X, Y) of each pixel in the BEV view can be calculated, which can also be understood as the coordinate values of the OXY plane in the vehicle coordinate system of the ego vehicle.

[0079] Specifically, taking the coordinates of a certain pixel point (e.g., pixel point a) in the vehicle coordinate system as (X, Y, Z) and the coordinates in the pixel coordinate system as (u, v) as an example, the coordinates of the pixel point a in the pixel coordinate system can be converted into the coordinates in the vehicle coordinate system according to the following formula (1):

[0080] wherein f x may represent the length of the focal length in the x-axis direction in the image coordinate system using pixels, f y may represent the length of the focal length in the y-axis direction in the image coordinate system using pixels, c x and c y may be used to represent the position of a pixel in the image coordinate system, with the unit of pixel (px), Z c is the value of the z-axis of the camera coordinate system corresponding to the pixel a. R and t represent the angle of rotation and the vector of translation of the camera coordinate system relative to the vehicle coordinate system, i.e., R and t are the extrinsic parameters of the camera.

[0081] The origin o of the camera coordinate system can be located at the center of the camera, and the x, y, and z axes are defined by the camera to follow the x, y, and z directions of the camera base. The origin O of the vehicle coordinate system can be located at the projection point of the center of the rear axle of the vehicle body on the ground, and the X, Y, and Z axes are the front direction, left direction, and vertically upward direction of the vehicle body, respectively.

[0082] Further, formula (1) can be simplified to formula (2):

[0083] Taking the pixels in the segmented slippery area as an example, which are in the Z=0 plane in the ego vehicle and vehicle coordinate system, formula (2) can be evolved into the following formula (3):

[0084] Further, formula (4) can be obtained according to formula (3), wherein inv(M') represents the inverse matrix of M'.

[0085] It can be understood that the segmentation result (i.e., the coordinates in the image coordinate system) can be converted to the BEV perspective by formula (4). For example, the slippery area shown in (a) of FIG. 5 is converted to the BEV perspective, and the image shown in (b) of FIG. 5 can be obtained, wherein the gray area is the distribution of the water in the BEV perspective.

[0086] In some implementations, the camera failure information of the camera can be determined according to the image captured by the camera outside the vehicle in time period 1, and the camera failure information can indicate the degree to which the lens of the camera is blocked by water stains. Specifically, the degree to which the lens of the camera is blocked by water stains can be measured by the image quality score, the water stain score, and the water stain segmentation ratio. For example, as shown in (a) of FIG. 6, the image captured by a camera outside the vehicle (for example, a rearview camera) can be evaluated according to multiple dimensions such as the degree of glare, the degree of translucent obstruction, the degree of water mist obstruction, and the degree of smoke obstruction. The translucent obstruction can be understood as being blocked by water stains, and the water mist obstruction can be water mist formed by small water droplets raised by rain. For example, the image quality score can be valued from 0 to 1, and the larger the value, the higher the image quality. In addition, the water stain score can also be determined according to the pixels associated with the water stains in the image. For example, the water stain score can be valued from 0 to 1, and the larger the value, the greater the influence of the water stains on the image. In actual implementation, the image quality score and the water stain score can be the average result of multiple continuous frames of images.

[0087] In some implementations, the image is divided into multiple grid areas (for example, as shown in (b) of FIG. 6), and the proportion of the area blocked by water stains to the total grid (or the water stain segmentation ratio) is determined. For example, the image can be divided into 32*18 grid areas, or the image can also be divided into other numbers of grid areas. Further, for multiple continuous frames of images, the proportion of the number of grid areas blocked by water stains in each frame of the multiple continuous frames of images to the total number of grid areas is determined, and the water stain segmentation ratio results corresponding to the multiple frames of images are smoothed and then averaged. The average value is the water stain segmentation ratio. For example, the water stain segmentation ratio can be valued from 0 to 1, and the higher the value of the water stain segmentation ratio, the greater the influence of the water stains on the image.

[0088] In some scenarios, only the image quality score cannot accurately reflect the degree to which the lens of the camera is blocked by water stains, because a low image quality score can be caused by smoke or other obstructions. In the case where the image quality score is low, the water stain score is high, and the water stain segmentation ratio is high, it can be determined that the image includes a large number of water stain-related pixels, or a large number of water stains are attached to the glass (for example, the camera lens) in the FOV of the camera.

[0089] S303, obtaining the detection result of the rain amount component.

[0090] In some implementations, the rain amount component includes components related to rain amount, such as a wiper and / or a rain sensor. The detection result of the rain amount component can be determined according to the wiper speed and the rain amount detected by the rain sensor. For example, the wiper speed can include 0-10 gears, where a higher gear represents a faster wiper swing speed. The rain amount detected by the rain sensor can include 0-13 gears, where a higher gear represents a larger rain amount detected by the rain sensor. Further, the wiper speed can be accumulated frame by frame to determine the rain amount level corresponding to the wiper; the detection result of the rain sensor can also be accumulated frame by frame to determine the rain amount level corresponding to the rain sensor.

[0091] For example, the count count1 of the current frame related to the wiper or the rain sensor can be determined according to the following formula (5):

[0092] wherein count0 is the count value of the previous frame, value1 is the quantized value of the rain sensor gear or the wiper gear corresponding to the current frame, and lower and upper are different threshold values. For example, the value of value1 can be determined according to the following formula (6):

[0093] wherein value' represents the rain sensor gear or the wiper gear of the current frame, and th represents the gear threshold value. For example, x can be a positive number and y can be a negative number. In addition, the values of x and y can be different for different gear threshold values.

[0094] It can be understood that the change of the count with the frame number shown in FIG. 7 can be obtained by formula (5) and formula (6). For example, as shown in (a) of FIG. 7, when the count value is greater than or equal to the count threshold value and the duration is greater than or equal to the frame number threshold value, the rain amount level corresponding to the wiper can be output as rain. As shown in (b) of FIG. 7, when the count value is greater than or equal to the count threshold value and the duration is greater than or equal to the frame number threshold value, the rain amount level corresponding to the rain sensor can be output as rain. It should be understood that the foregoing formula (5) and formula (6) are only exemplary descriptions, and in actual implementation, more gear threshold values and count calculation methods can be set to determine more rain amount levels, such as rain, light rain, moderate rain, heavy rain, etc.

[0095] S304, determining the rain amount level according to the perception information processing result and the detection result of the rain amount component.

[0096] In some implementations, the rain amount level can be determined in any one of the following three ways:

[0097] In a first manner, the perception information processing result and the detection result of the rain amount component can be input into a rain amount decision model. The model can be based on a lightweight decision tree algorithm in a gradient boosting framework.

[0098] It can be understood that the decision tree algorithm is generally a process of recursively selecting an optimal feature and classifying the data to be processed according to the feature, so that each subset has the best classification. For example, in the process of training the rain amount decision model, the aforementioned perception information processing result and the detection result of the rain amount component can be placed in the root node, and a feature that can distinguish the data in the root node is selected as the splitting criterion of the current root node according to a certain index (such as information gain, information gain ratio, Gini coefficient, etc.). Further, the data set in the root node is divided into subsets according to the aforementioned splitting criterion, and each subset corresponds to a value that can best distinguish the data in the root node. If the subset can be classified basically correctly, a leaf node is constructed, and the subset is assigned to the corresponding leaf node. If the subset cannot be classified basically correctly, a new feature that can distinguish the data in the root node is selected as the splitting criterion to continue to divide the data set, and the corresponding leaf node is constructed. The above process is repeated for the subset, and a subtree is recursively constructed until a stop condition is met (such as reaching a maximum tree depth, the leaf node data amount being less than a minimum sample number, etc.). When further division is not possible (basic correct classification or no suitable feature), a leaf node is generated and a classification or regression result is assigned, that is, each subset is assigned to a leaf node, that is, each has a clear class, and thus a decision tree is generated. In an example, in order to improve computational efficiency and reduce memory overhead, when determining the splitting criterion, continuous floating point feature values can be discretized into k integers, and a histogram with a width of k is constructed. When traversing the data, the discretized value is used as an index to accumulate statistical quantities in the histogram. After traversing the data once, the histogram accumulates the required statistical quantities, and then the optimal splitting point is found by traversing the histogram according to the discrete values. In another example, in order to improve the construction speed of the decision tree, a leaf node with the largest splitting gain can be found from all existing leaf nodes, and then split, and the cycle is repeated. In the case of the same number of splits, this method can reduce more errors and obtain better accuracy. At the same time, in order to prevent overfitting, a depth limit (such as a set of parameters) can be set in advance to prevent the generation of a too deep decision tree. For example, when the training does not show significant accuracy improvement within a certain number of rounds, the training should be stopped in time.

[0099] Exemplarily, in order to optimize the parameter configuration of the rainfall decision model, a grid search method is adopted, a set of hyperparameter space is set in advance, all parameter combinations in the hyperparameter space are exhausted, and a more optimal parameter configuration can be obtained through reasonable evaluation criteria. The aforementioned evaluation criteria can be determined by a cross-validation method, for example, by dividing the data set multiple times in different degrees, verifying the f1-score of each division, and finally averaging the results of multiple divisions, a more accurate evaluation criterion can be obtained.

[0100] Exemplarily, in order to improve the accuracy of the rainfall decision model, in the model training process, the bagging strategy is introduced by using the idea of ensemble learning. Specifically, multiple subsets are randomly extracted from the subset of training data, and the size of any two subsets is the same (such as the same amount of data included). Each subset is input into the rainfall decision module, and a base model is independently trained based on each subset. The prediction results of multiple base models are combined through voting or averaging to form the final prediction result.

[0101] Method two: The perception information processing result and the detection result of the rainfall component can also be quantified, and then the rainfall grade P is determined according to the following formula (7): P = a * P1 + b * P2; (7)

[0102] Wherein, P1 represents the rainfall grade determined according to the perception information processing result, P2 represents the rainfall grade determined according to the detection result of the rainfall component, and a and b represent the weights of P1 and P2 respectively. Exemplarily, P1 can take a value of one of 0 to 1, and P2 can also take a value of one of 0 to 1. The larger the value, the greater the rainfall. a and b take a value of one of 0 to 1, and the sum of a and b is 1. More specifically, P1 can be the result of quantifying the visual rainfall grade, the road wetness degree, and the camera failure information. For example, P1 can satisfy the following formula (8):

[0103] Wherein, Pa and Pb represent the quantized value of the visual rainfall grade and the quantized value of the road wetness degree respectively, and a1 and a2 represent the weight of Pa and Pb in P1 respectively. Pci to Pei represent the image quality score, water stain score and water stain segmentation proportion of camera i respectively, and a3i, a4i and a5i represent the weight of Pci, Pdi and Pei respectively, and n is the total number of cameras detected by the camera failure detection module, which can be an integer greater than or equal to 1. For example, the value of Pa can be 0 to 1, when Pa is 0, it represents no rain, and when Pa is 1, it represents heavy rain. The value of Pb can be 0 to 1, when the value of Pb is 0, it represents a dry road surface and no water on the road surface, and when the value of Pb is not 0, it represents the proportion of the water area on the road surface to the overall area of the road surface. The overall area of the road surface is the area of the road surface within a certain distance (such as 30 meters, or 20 meters, or other distances) in front of the vehicle included in the image.

[0104] It can be understood that the determined rainfall grade P can also have a value of a number between 0 and 1, and the larger the value, the larger the rainfall grade.

[0105] Method three: The rainfall grade can also be determined according to the result output by the camera failure detection module combined with the detection result of the rainfall component.

[0106] The camera failure detection module can deduce the image outside the vehicle collected by the camera arranged in the vehicle to obtain a deduction result, which can include an image quality score and an image area quality score. The image area quality score can indicate the quality score of each area in the image. The areas include areas that can be wiped by the wiper blade when the wiper blade swings and areas that cannot be wiped by the wiper blade when the wiper blade swings. The quality score of each area can be evaluated according to the evaluation method of the image quality score. The value of the quality score of each area can be any value between 0 and 1. The greater the value, the better the image quality of the area. As shown in FIG. 8, the result output by the camera failure detection module is input to the post-processing module to exclude the influence of the wiper blade on the image quality judgment accuracy. Specifically, the post-processing module buffers at least the deduction results of the past three frames of images. The post-processing module determines whether there is an area 1 in the image that can be wiped by the wiper blade swing and has an area image quality score less than or equal to a threshold 1 from the deduction result. The area 1 can be understood as a pixel area corresponding to the wiper blade shielding part. If the area 1 exists in the intermediate frame, the area 1 of the intermediate frame is compared with the front and rear frames to determine whether there is an overlapping part. If the area of the area 1 of the intermediate frame is greater than or equal to a pixel threshold 1, and there is no overlapping part between the area 1 of the intermediate frame and the front and rear frames, it is considered that the intermediate frame is shielded by the wiper. For the image shielded by the wiper, the deduction result of the image is deleted, or the deduction result of the last frame of image is input to the feature extraction module.

[0107] For example, the threshold 1 can be 0.1-0.2, or other values. The pixel threshold 1 can be 10-15px, or other values. In addition, the modules in the dashed box shown in FIG. 8 can be arranged in the rain detection module 240 shown in FIG. 2.

[0108] For example, as shown in FIG. 9, a certain area of the intermediate frame includes pixels corresponding to the wiper blade. The pixel area of the wiper blade can be regarded as an example of the area 1. As can be seen, the previous frame and the next frame do not have pixels corresponding to the wiper blade. Therefore, it can be determined that the intermediate frame is shielded by the wiper.

[0109] Exemplarily, the post-processing module sends the inference result of the image to the feature extraction module at a frequency of p Hz, and in addition, the detection result of the rain component is also input to the feature extraction module at a frequency of p Hz. For example, the aforementioned p can be 10, i.e., the time interval between adjacent two frames is 0.1 s. The feature extraction module caches n frames of results, each of which includes the inference result of a frame of image, or each of which can further include the detection result of the rain component associated with the inference result of the frame of image. It should be understood that the n frames of results are updated as the inference result of the post-processing module is input. Further, the feature extraction module performs feature extraction at a frequency of q Hz, and then inputs the extracted features to the rain decision model to obtain the inferred rain grade. The training method of the rain decision model can refer to the description in the foregoing embodiments, which will not be described here again.

[0110] Exemplarily, the aforementioned p can be any value in 10-15, or p can also be other values; the aforementioned q can be any value in 1-5, or q can also be other values less than p; n can be any value in 50-100, or can also be other values.

[0111] In some implementations, the features extracted by the feature extraction module can include at least one of: the average value of the image quality score, the mean square error of the image quality score, the frequency of the image quality score being greater than or equal to the score threshold 1, the number of frames during which the image quality score is greater than or equal to the score threshold 1, the frequency of the image quality score being less than or equal to the score threshold 2, and the number of frames during which the image quality score is less than or equal to the score threshold 2.

[0112] Exemplarily, the average value and the mean square error can be calculated according to the data within the n-frame sliding window, i.e., for the average value and the mean square error of the image quality score, the average value and the mean square error of the image quality scores corresponding to the n frames of results cached in the current feature extraction module are directly calculated.

[0113] Exemplarily, the score threshold 1 can be any value in 0.8, 0.9, 0.95, or can also be other values, and the frequency of the image quality score being greater than or equal to the score threshold 1 can be: the number of times that the image quality score in the n-frame result is detected to appear from being lower than the score threshold 1 to being greater than or equal to the score threshold 1, or the number of times that the image quality score in the n-frame result is greater than or equal to the score threshold 1. For example, the score threshold 1 is 0.8, and the image quality scores of the 11th frame to the 20th frame in the n-frame result are 0.5, 0.5, 0.9, 0.85, 0.85, 0.8, 0.7, 0.6, 0.5, and 0.5, respectively, then the number of times that the image quality score in the 11th frame to the 20th frame appears from being lower than the score threshold 1 to being greater than or equal to the score threshold 1 is 1, and the number of times that the image quality score in the 11th frame to the 20th frame is greater than or equal to the score threshold 1 is 4. Further, the number of frames in which the image quality score in the 11th frame to the 20th frame is greater than or equal to the score threshold 1 is 4. In actual implementation, the number of frames in which the image quality score is greater than or equal to the score threshold 1 can be an average value corresponding to the detection result of n frames or more frames.

[0114] Exemplarily, the score threshold 2 can be any value in 0.4 to 0.6, or can also be other values, and the frequency of the image quality score being less than or equal to the score threshold 2 can be: the number of times that the image quality score in the n-frame result is detected to appear from being higher than the score threshold 2 to being less than or equal to the score threshold 2, or the number of times that the image quality score in the n-frame result is less than or equal to the score threshold 2. The more specific meaning of the frequency of the image quality score being less than or equal to the score threshold 2 can refer to the description in the previous paragraph, which is not repeated here.

[0115] In some implementations, the foregoing score threshold 1 and / or score threshold 2 can also be a group of thresholds, and then a group of features associated with the score threshold 1 and / or score threshold 2 can be obtained according to the score threshold 1. Taking the score threshold 1 as th1 = {0.8, 0.9, 0.95} and the frequency of the image quality score being greater than or equal to the score threshold 1 indicating the number of times that the image quality score in the n-frame result is greater than or equal to the score threshold 1 as an example, if the number of times that the image quality score in the n-frame result is greater than or equal to 0.8, 0.9, and 0.95 is 0.6n, 0.5n, and 0.4n, respectively, then the features associated with the score threshold 1 include three groups of features, which are 0.6n, 0.5n, and 0.4n, respectively.

[0116] In actual implementation, the n-frame result can also be processed by a convolution kernel to obtain higher-dimensional feature information. For example, a convolution kernel is set as weights = [0.5, 1, 0.5], and the inner product of weights and the image quality scores in the n-frame result is obtained to obtain the feature indicating the image quality.

[0117] It should be understood that the foregoing is described by taking the image quality score in n frames of results as an example. In actual implementation, the gear (or swing speed) of the wiper and other features of the image can also be extracted, and then the extracted multi-dimensional features are input into the rain amount decision model to obtain the detection result of the rain amount level.

[0118] In some implementations, in addition to processing the foregoing multi-dimensional features based on the rain amount decision model to obtain the rain amount level, each feature in the foregoing multi-dimensional features can be quantified, and then the rain amount level is determined according to the quantized value of each feature. For example, the rain amount level P' can be determined based on the following formula (9): P' = a' * p1 + b' * p2 + c' * p3 + d' * p4 + e' * p5 + f' * p6 + g' * p7; (9)

[0119] wherein p1 to p7 respectively represent the average value of the image quality score, the mean square error of the image quality score, the frequency of the image quality score being greater than or equal to the score threshold 1, the frame number of the image quality score being greater than or equal to the score threshold 1, the frequency of the image quality score being less than or equal to the score threshold 2, the frame number of the image quality score being less than or equal to the score threshold 2, and the gear of the rain amount component, a' to g' respectively represent the weight of p1 to p7. The values of p1 to p7 can be 0-1 respectively. For p1, the greater the value, the lower the average value of the image quality score. For p2, the greater the value, the lower the mean square error of the image quality score. For p3, the greater the value, the lower the frequency of the image quality score being greater than or equal to the score threshold 1. For p4, the greater the value, the fewer the frame number of the image quality score being greater than or equal to the score threshold 1. For p5, the greater the value, the higher the frequency of the image quality score being less than or equal to the score threshold 2. For p6, the greater the value, the more the frame number of the image quality score being less than or equal to the score threshold 2. The values of a' to g' can be 0-1 respectively, and the sum of a' to g' is 1. That is, the value of P' is 0-1, and the greater the value, the higher the rain amount level.

[0120] In actual implementation, the rain intensity level can be determined according to any one of the preceding modes 1 to 3. Alternatively, one of the modes can be selected according to the detection result of the rain amount component. For example, when the gear of the windscreen wiper is greater than or equal to a preset gear threshold and the duration is greater than or equal to a duration threshold, and / or the gear of the rain sensor is greater than or equal to a preset threshold and the duration is greater than or equal to a duration threshold, the rain intensity is detected by mode 3; otherwise, the rain intensity is detected by mode 1 or mode 2. For example, when the gear of the windscreen wiper is 0 to 10 gears, the preset gear threshold can be any one of 6 to 8 gears, or can be other values; for example, when the gear of the rain sensor is 0 to 13 gears, the preset threshold can be any one of 6 to 9 gears, or can be other values; the duration threshold can be 3 to 5 minutes, or can be other values.

[0121] In some implementations, after determining the rain intensity level, if the rain intensity level is greater than or equal to a level threshold, the vehicle is controlled to exit the intelligent driving function, or the use of the intelligent driving function is limited. The intelligent driving function can include, but is not limited to, the following functions that affect driving safety: auto parking assist (APA), auto valet parking (AVP), adaptive cruise control (ACC), lane cruise control (LCC), navigate cruise assist (NCA), front vehicle collision warning, lane departure warning, lane keeping assist, rear vehicle collision warning. The NCA function refers to a function of controlling the vehicle to travel to a destination according to a navigation route, and being able to control the vehicle to pass through an intersection, change lanes, and change gears according to road information such as traffic lights.

[0122] For example, if the rain intensity level includes no rain, light rain, moderate rain, heavy rain, and heavy rain, the level threshold can be heavy rain or heavy rain; if the rain intensity level is a quantitative value, such as quantifying the rain intensity level as a parameter with any value from 0 to 1, and the rain intensity increases as the value increases, the level threshold can be any value from 0.6 to 0.8, or the level threshold can be other values.

[0123] FIG. 10 shows a schematic flowchart of a rain intensity detection method provided by an embodiment of the application, which can be executed by the computing platform 150 shown in FIG. 1, or by the system shown in FIG. 2. The method can include:

[0124] S1010, acquire multi-source information, the multi-source information at least including: multi-frame images collected by at least one camera device of the vehicle, and rain component information, the rain component information indicating a gear position of a rain wiper of the vehicle and / or a rain amount detected by a rain sensor of the vehicle.

[0125] Exemplarily, the at least one camera device can include the at least one camera device in the method 300, and the rain sensor can include the rain sensor in the method 300.

[0126] S1020, determine camera failure information according to the multi-frame images, the camera failure information indicating an influence degree of the rain amount on an image definition of each of the at least one camera device.

[0127] Exemplarily, the camera failure information can include the image quality score, the water stain score and the water stain segmentation ratio in the method 300; or, the camera failure information can also only include the image quality score and the image region quality score in the method 300.

[0128] S1030, determine a rain amount grade according to at least the camera failure information and the rain component information.

[0129] In some implementations, the camera failure information includes an image quality score, a water stain score and a water stain segmentation ratio corresponding to the image collected by each camera device, wherein the image quality score indicates a quality of the image, the water stain score indicates a probability that the quality of the image is affected by water stains, and the water stain segmentation ratio indicates a proportion of pixels associated with water stains in the image relative to all pixels; and S1030 can be refined as: determining the rain amount grade according to the image quality score, the water stain score, the water stain segmentation ratio and the rain component information. More specifically, a method of determining the rain amount grade according to the image quality score, the water stain score, the water stain segmentation ratio and the rain component information can refer to the description in the first mode or the second mode in the method 300, which will not be described here again.

[0130] In some implementations, the multiple frames of images include a first set of images captured by the first camera, the camera failure information includes an image quality score of each frame of image in the first set of images, the image quality score indicating a quality of each frame of image; and determining the rain intensity level according to at least the camera failure information and the rain component information includes: determining at least one feature according to the image quality score of each frame of image, the at least one feature including: a score mean, a standard deviation, a first frequency feature, and a second frequency feature; wherein the score mean is an average of the image quality scores corresponding to the first set of images, the standard deviation is a standard deviation of the image quality scores corresponding to the first set of images, the first frequency feature is a frequency and / or a number of consecutive frames that the image quality score is greater than or equal to a first score threshold, and the second frequency feature is a frequency and / or a number of consecutive frames that the image quality score is less than or equal to a second score threshold; and determining the rain intensity level according to at least the at least one feature and the rain component information.

[0131] For example, the first score threshold can be the aforementioned score threshold 1, and the second score threshold can be the aforementioned score threshold 2, or the score threshold 1 and the score threshold 2 can also be other numerical values.

[0132] In some implementations, the field of view angle coverage area of the first camera covers an area of the windshield wiped by the wiper, and the first frequency feature and / or the second frequency feature is associated with a gear position of the wiper.

[0133] For example, the first camera can include a front-view camera disposed inside the windshield. For a specific implementation of determining the at least one feature according to the quality score of each frame of image, reference can be made to the description in Mode 3 of the method 300, which will not be repeated here.

[0134] In some implementations, each frame of image includes a first pixel area and a second pixel area, the first pixel area includes pixels corresponding to all or part of the area of the windshield wiped by the wiper, and the second pixel area includes pixels corresponding to an area other than the area of the windshield wiped by the wiper; and a region quality score of each frame of image is greater than or equal to a third score threshold; wherein the region quality score indicates an image quality of the first pixel area. For example, the third score threshold can be the aforementioned threshold 1, or can also be other numerical values. For a specific implementation of discarding the image including the pixels corresponding to the wiper, reference can be made to the description in Mode 3 of the method 300, which will not be repeated here.

[0135] In some implementations, the multiple frames of images include a second set of images captured by a second camera, the second camera being configured to capture images in front of the vehicle, and the method further includes: determining a visual rain intensity level according to the second set of images; and determining the rain intensity level includes: determining the rain intensity level according to at least the camera failure information, the rain component information, and the visual rain intensity level. For example, the second camera can include a front-view wide-angle camera.

[0136] In some implementations, the multiple frames of images include a third set of images captured by a third camera device, the third camera device being configured to acquire images in front of the vehicle, each frame of image in the third set of images including pixels corresponding to a road surface, and the method further includes: determining a road surface wetness level based on the third set of images; and determining the rain intensity level based on the camera failure information, the rain component information, and the road surface wetness level. Exemplarily, the third camera device can include a front long-focus camera device.

[0137] Exemplarily, the method of determining the visual rain intensity level can refer to the description in S302; the road surface wetness level can be determined according to a proportion of a road surface water area to a total area of the road surface, and the higher the proportion, the higher the road surface wetness level. A more specific method of determining the road surface wetness level can refer to the description in S302, which will not be repeated here. In addition, the specific implementation of determining the rain intensity level based on the visual rain intensity level and / or the road surface wetness level, and the camera failure information and the rain component information, can refer to the description in the aforementioned manner one or manner two, which will not be repeated here.

[0138] The rain intensity detection method provided by the embodiments of the present application determines the rain intensity level based on multiple source information including images captured by a camera device and information of a rain component, which helps to improve the detection accuracy of the rain intensity level and improve the applicability of the rain intensity detection method in different scenarios. When the rain intensity level is greater than or equal to a rain intensity threshold, the intelligent driving function of the vehicle can be limited to be enabled, which helps to improve the driving safety of the vehicle.

[0139] In various embodiments of the present application, the terms and / or descriptions of various embodiments are consistent and can be mutually referred to if there is no special description and logical conflict. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0140] The method provided by the embodiments of the present application is described in detail above in combination with FIGS. 1 to 10. The device provided by the embodiments of the present application will be described in detail below in combination with FIGS. 11 and 12. It should be understood that the description of the device embodiments corresponds to the description of the method embodiments, and therefore, the content not described in detail can refer to the method embodiments described above, which will not be repeated here for brevity.

[0141] FIG. 11 shows a schematic block diagram of the rain detection device 2000 provided by the embodiments of the present application, which can include units for performing the methods described in the foregoing embodiments. Each unit in the device 2000 is configured to perform a corresponding function to enable the device to implement the methods performed by the device. The device 2000 includes an obtaining unit 2010, which can be configured to implement a corresponding data obtaining or transmitting function. The device 2000 further includes a processing unit 2020, which can be configured to implement a corresponding processing function.

[0142] Optionally, the device 2000 further includes a storage unit, which can be configured to store instructions and / or data. The processing unit 2020 can read the instructions and / or data in the storage unit to cause the device to implement the related actions in the foregoing various method embodiments.

[0143] It should be understood that the specific process by which each unit performs the corresponding steps described above has been described in the foregoing method embodiments, and thus will not be described here for brevity.

[0144] It should also be understood that the device 2000 herein is embodied in the form of functional units. The term “module” or “unit” herein can refer to an application-specific ASIC, an electronic circuit, a processor (shared, dedicated, or group) and memory for executing one or more software or firmware programs, a combined logic circuit and a suitable component for supporting the described functions.

[0145] The device of each of the above-described solutions has the function of implementing the corresponding steps performed by the computing platform 150 in the above-described methods. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions; for example, the obtaining unit 2010 can be replaced by a transceiver, and other units, such as the processing unit, can be replaced by a processor, for performing the related processing operations in the various method embodiments.

[0146] For example, the obtaining unit 2010 and the processing unit 2020 can be arranged in the rain detection module 240, and the operations performed by the obtaining unit 2010 and the processing unit 2020 described above can be performed by one processor, or can also be performed by different processors. In a specific implementation process, the one or more processors described above can be processors arranged in the computing platform 150 of the vehicle; or the device 2000 described above can be a chip arranged in the vehicle.

[0147] The processor involved in the present application is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), a digital signal processor (DSP), etc. In another implementation, the processor can implement certain functions through the logical relationship of hardware circuits, which is fixed or can be reconfigured. For example, the processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field programmable gate array (FPGA). In a reconfigurable hardware circuit, the processor loads a configuration document to implement the hardware circuit configuration process, which can be understood as the process of the processor loading instructions to implement related functions. In addition, the processor can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. In some implementations, the computing platform can also include a memory for storing instructions, and the processor can call the instructions in the memory to implement corresponding functions.

[0148] In the implementation process, all or part of the units in the above device can be integrated together or independently implemented. In one implementation, these units are integrated together to implement a system-on-a-chip (SoC).

[0149] Fig. 12 is another schematic block diagram of the rain detection apparatus provided in the embodiments of the present application. The apparatus 2100 shown in Fig. 12 can include a processor 2110, a transceiver 2120 and a memory 2130. The processor 2110, the transceiver 2120 and the memory 2130 are connected through internal connection paths. The memory 2130 is configured to store instructions, and the processor 2110 is configured to execute the instructions stored in the memory 2130 to implement the methods in the above embodiments. Alternatively, the memory 2130 can be coupled to the processor 2110 through an interface, or integrated with the processor 2110.

[0150] It should be noted that the transceiver 2120 can include, but is not limited to, a transceiving device such as an input / output interface, to implement the communication between the apparatus 2100 and other devices or communication networks.

[0151] The memory 2130 can be a volatile memory and / or a non-volatile memory. The non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM). For example, the RAM can be used as an external cache. By way of example and not limitation, the RAM includes the following various forms: a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a synchlink DRAM (SLDRAM), and a direct rambus RAM (DR RAM).

[0152] The transceiver 2120 uses a transceiving device such as, but not limited to, a transceiver, to implement the communication between the apparatus 2100 and other devices or communication networks, to receive / send data / information used to implement the methods in the above embodiments.

[0153] The embodiment of the present application further provides an intelligent driving device, which comprises the device 2000 or the device 2100 in the above embodiment.

[0154] The embodiment of the present application further provides a computer program product, which comprises computer program codes, and when the computer program codes are run on a computer, the computer program codes make the computer implement the method in the above embodiment of the present application.

[0155] The embodiment of the present application further provides a computer readable storage medium, which stores computer instructions, and when the computer instructions are run on a computer, the computer instructions make the computer implement the method in the above embodiment of the present application.

[0156] The embodiment of the present application further provides a chip, which comprises a circuit, and is used for executing the method in the above embodiment of the present application.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, the device and the unit described above can refer to the corresponding process in the foregoing method embodiments, and will not be described here.

[0158] In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" in this paper is a description of the association relationship of the associated object, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0159] In the embodiments of the present application, the prefix words such as "first", "second" are only used to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of ordinal words such as ordinal words in the embodiments of the present application does not constitute a limitation on the described objects, and the description of the described objects should refer to the description of the context in the claims or embodiments, and should not constitute redundant limitations because of the use of such prefix words.

[0160] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the division of the above-described device embodiment is merely an example, and there can be other division manners. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0161] In each embodiment of the present application, the terms and / or descriptions between different embodiments are consistent and can be mutually referred to if there is no special description and logical conflict, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0162] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0163] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can be a physically independent unit, or two or more units can be integrated into a unit.

[0164] The above is merely specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A rain detection method, characterized by, The method comprises: obtaining multi-source information, the multi-source information at least comprising: a plurality of images collected by at least one camera of a vehicle, and rain amount component information indicating a gear position of a rain wiper of the vehicle and / or a rain amount detected by a rain sensor of the vehicle; determining camera failure information according to the plurality of images, the camera failure information indicating an influence degree of rain amount on image clarity of each image collected by each camera of the at least one camera; determining a rain amount level according to at least the camera failure information and the rain amount component information.

2. The method of claim 1, wherein, The camera failure information comprises an image quality score, a water stain score and a water stain segmentation ratio corresponding to each image collected by each camera, wherein the image quality score indicates the quality of the image, the water stain score indicates a probability that the quality of the image is affected by water stains, and the water stain segmentation ratio indicates a proportion of water stain-related pixels in all pixels in the image. The determination of the rain amount level comprises: determining the rain amount level according to the image quality score, the water stain score, the water stain segmentation ratio and the rain amount component information.

3. The method of claim 1, wherein, The plurality of images comprises a first group of images collected by a first camera, and the camera failure information comprises an image quality score of each image in the first group of images, the image quality score indicating the quality of each image. The determination of the rain amount level according to at least the camera failure information and the rain amount component information comprises: determining at least one feature according to the image quality score of each image, the at least one feature comprising: a score mean value, a standard deviation, a first frequency feature and a second frequency feature; wherein the score mean value is an average value of the image quality scores corresponding to the first group of images, the standard deviation is a standard deviation of the image quality scores corresponding to the first group of images, the first frequency feature is a frequency and / or a number of continuous frames that the image quality score is greater than or equal to a first score threshold, and the second frequency feature is a frequency and / or a number of continuous frames that the image quality score is greater than or equal to a score interval; determining the rain amount level according to the at least one feature and the rain amount component information.

4. The method of claim 3, wherein, A field of view angle coverage area of the first camera covers a windshield area wiped by the rain wiper, and the first frequency feature and / or the second frequency feature are associated with the gear position of the rain wiper.

5. The method according to claim 3 or 4, characterized in that, Each image comprises a first pixel area and a second pixel area, the first pixel area comprises pixels corresponding to all or part of the windshield area wiped by the rain wiper, and the second pixel area comprises pixels corresponding to an area other than the windshield area wiped by the rain wiper. Furthermore, a region quality score of each image is greater than or equal to a third score threshold, wherein the region quality score indicates the image quality of the first pixel area.

6. The method according to any one of claims 1 to 5, characterized in that, The plurality of images comprises a second group of images collected by a second camera, the second camera being configured to collect images in front of the vehicle, and the method further comprises: determining a visual rain amount level according to the second group of images; The determination of the rain amount level comprises: According to the camera failure information, the rain amount component information, and the visual rain amount grade, a rain amount grade is determined.

7. The method of claim 6, wherein, The second camera is a wide-angle camera.

8. The method according to any one of claims 1 to 7, characterized in that, The multiple frames of images include a third set of images captured by a third camera, the third camera being configured to acquire images in front of the vehicle, each frame of the third set of images including pixels corresponding to a road surface, and the method further includes: According to the third set of images, a road surface wetness degree is determined. The determination of the rain amount grade includes: According to the camera failure information, the rain amount component information, and the road surface wetness degree, the rain amount grade is determined.

9. The method of claim 8, wherein, The third camera is a long-focus camera.

10. A rain detection device, characterized by It includes: An acquisition unit is configured to acquire multiple-source information, the multiple-source information including at least multiple frames of images captured by at least one camera of a vehicle and rain amount component information indicating a gear position of a windshield wiper of the vehicle and / or a rain amount detected by a rain sensor of the vehicle; A processing unit is configured to determine camera failure information according to the multiple frames of images, the camera failure information indicating an influence degree of rain amount on image definition of each image captured by each camera of the at least one camera; The processing unit is further configured to determine a rain amount grade according to at least the camera failure information and the rain amount component information.

11. The apparatus of claim 10, wherein, The camera failure information includes an image quality score, a water stain score, and a water stain segmentation ratio corresponding to each image captured by each camera, wherein the image quality score indicates a quality of an image, the water stain score indicates a probability of the quality of the image being affected by a water stain, and the water stain segmentation ratio indicates a proportion of pixels associated with the water stain in all pixels in the image; The processing unit is configured to: Determine the rain amount grade according to the image quality score, the water stain score, the water stain segmentation ratio, and the rain amount component information.

12. The apparatus of claim 10, wherein, The multiple frames of images include a first set of images captured by a first camera, and the camera failure information includes an image quality score of each frame of the first set of images, the image quality score indicating a quality of each frame of the first set of images; The processing unit is configured to: Determine at least one feature according to the image quality score of each frame, the at least one feature including a score mean value, a standard deviation, a first frequency feature, and a second frequency feature; The score mean value is an average value of the image quality scores corresponding to the first set of images, the standard deviation is a standard deviation of the image quality scores corresponding to the first set of images, the first frequency feature is a frequency and / or a number of continuous frames at which the image quality score is greater than or equal to a first score threshold, and the second frequency feature is a frequency and / or a number of continuous frames at which the image quality score is less than or equal to a second score threshold; Determine the rain amount grade according to the at least one feature and the rain amount component information.

13. The apparatus of claim 12, wherein, A field of view angle coverage area of the first camera covers a windshield area wiped by the windshield wiper, and the first frequency feature and / or the second frequency feature are associated with the gear position of the windshield wiper.

14. The apparatus of claim 12 or 13, wherein, The each frame image comprises a first pixel region and a second pixel region, the first pixel region comprises pixels corresponding to all or part of the windshield region wiped by the wiper, and the second pixel region comprises pixels corresponding to a region other than the windshield region wiped by the wiper. The region quality score of the each frame image is greater than or equal to a third score threshold; wherein the region quality score indicates the image quality of the first pixel region.

15. The apparatus of any one of claims 10-14, wherein, The plurality of frame images comprise a second group of images captured by a second camera device, the second camera device is configured to acquire images in front of the vehicle, and the processing unit is further configured to: determine a visual rain amount level according to the second group of images; determine the rain amount level according to the camera failure information, the rain amount component information, and the visual rain amount level.

16. The apparatus of claim 15, wherein, The second camera device is a wide-angle camera device.

17. The apparatus of any one of claims 10-16, wherein, The plurality of frame images comprise a third group of images captured by a third camera device, the third camera device is configured to acquire images in front of the vehicle, and each frame image in the third group of images comprises pixels corresponding to a road surface, and the processing unit is further configured to: determine a road surface wetness level according to the third group of images; determine the rain amount level according to the camera failure information, the rain amount component information, and the road surface wetness level.

18. The apparatus of claim 17, wherein, The third camera device is a long-focus camera device.

19. A rain detection device, characterized by comprising: a processor configured to execute a computer program stored in a memory, so that the apparatus performs the method according to any one of claims 1 to 9.

20. The apparatus of claim 19, wherein, The apparatus further comprises the memory.

21. A computer-readable storage medium, characterized in that, instructions stored thereon, which when executed by a processor, implement the method according to any one of claims 1 to 9.

22. A chip, characterized by The chip comprises a circuit configured to execute the method according to any one of claims 1 to 9.

23. A computer program product, characterised in that, The computer program product comprises computer program code which, when executed by a processor, implements the method according to any one of claims 1 to 9.

24. A vehicle characterized by comprising: comprising the apparatus according to any one of claims 10 to 20, or the computer readable storage medium according to claim 21, or the chip according to claim 22, or the vehicle is loaded with the computer program product according to claim 23.

Images