Rainfall amount identification method, electronic device, and program product

Abstract

This disclosure provides a rainfall amount identification method, electronic device, and program product, relating to the field of intelligent driving technology. The method includes: acquiring audio signals collected by an onboard audio sensor, images collected by an onboard camera, and point cloud data collected by an onboard radar sensor; extracting rain audio features from the audio signals, rain image features from the images, and rain point cloud features from the point cloud data; fusing the rain audio features, rain image features, and rain point cloud features to determine a rainfall level; wherein the rainfall level reflects the amount of rainfall. This disclosure aims to solve the problems of poor environmental adaptability, low identification accuracy, and insufficient robustness in existing rainfall amount identification technologies.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent driving technology, specifically to a method for identifying rainfall, electronic equipment, and software products. Background Technology

[0002] Rainfall recognition is a core foundational function of vehicle advanced driver assistance systems (ADAS) and the perception layer of autonomous driving. Its performance directly determines the reliability and safety of downstream functions such as automatic wiper control, intelligent headlight adjustment, driving safety warnings, and powertrain torque correction.

[0003] As the level of automotive intelligence evolves to L3 and above, stringent requirements are placed on the adaptability, real-time performance, and accuracy of rainwater volume recognition under all operating conditions. It needs to stably output accurate rainwater volume levels in extreme temperature ranges of -40℃ to 85℃, driving speed ranges of 0 to 130 km / h, and complex environments such as strong light, backlight, low illumination at night, heavy rain, and heavy snow.

[0004] Current mainstream solutions for rainwater identification rely on single sensors or simple pairwise fusion, which generally suffer from poor environmental adaptability, low recognition accuracy, and insufficient robustness, making it difficult to meet the needs of advanced intelligent driving. Summary of the Invention

[0005] In view of this, the present disclosure provides a method, electronic device and program product for identifying rainfall volume, in order to solve the problems of poor environmental adaptability, low identification accuracy and insufficient robustness of existing rainfall volume identification methods.

[0006] In a first aspect, this disclosure provides a method for identifying rainfall, including: Acquire audio signals collected by the vehicle's audio sensor, images collected by the vehicle's camera, and point cloud data collected by the vehicle's radar sensor; Rain audio features are extracted from the audio signal, rain image features are extracted from the image, and rain point cloud features are extracted from the point cloud data; The rainfall level is determined by integrating the rain audio features, the rain image features, and the rain point cloud features; wherein the rainfall level is used to reflect the amount of rainfall.

[0007] Secondly, this disclosure provides an electronic device, including: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores at least one computer program that can be executed by the at least one processor, the at least one computer program being executed by the at least one processor to enable the at least one processor to perform the rainfall identification method as described in the first aspect.

[0008] Thirdly, this disclosure provides a computer program product, which includes a computer program that, when run in a processor, implements the rainfall identification method described in the first aspect.

[0009] The embodiments provided in this disclosure acquire audio signals from an onboard audio sensor, images from an onboard camera, and point cloud data from an onboard radar sensor. Rainfall audio features, rainfall image features, and rainfall point cloud features are extracted from the audio signals, images, and point cloud data, respectively. These three sensor features, derived from different physical principles, are then fused to determine the rainfall level. This fully leverages the complementary advantages of radar's all-weather penetration, audio's reflection of dynamic rainfall intensity, and image's capture of visual morphology. Thus, this method overcomes the performance bottleneck of a single sensor, stably outputting accurate rainfall levels in complex combined scenarios, effectively reducing the false recognition rate, and improving the overall robustness and all-weather adaptability of rainfall identification. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0011] Figure 1 The diagram shown is a schematic representation of the rainwater volume identification system in an embodiment of this disclosure.

[0012] Figure 2 The diagram shows the distribution of the sensor modules in the vehicle according to an embodiment of this disclosure.

[0013] Figure 3 The diagram shown is a flowchart of the rainfall identification method in an embodiment of this disclosure.

[0014] Figure 4 The diagram shown illustrates the decision-making process of the decision fusion submodule in this embodiment.

[0015] Figure 5 The diagram shown is a schematic diagram of the process of obtaining the fused feature vector in an embodiment of this disclosure.

[0016] Figure 6 The diagram shown illustrates the process of adaptively adjusting the weights of each modality and outputting the confidence score in an embodiment of this disclosure.

[0017] Figure 7 The diagram shown is a structural schematic of an electronic device in an embodiment of this disclosure. Detailed Implementation

[0018] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0019] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0020] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0021] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0022] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0023] Among related technologies, rainfall identification technology can be divided into two categories, and each type of solution has significant technical shortcomings.

[0024] The first category is traditional mechanical or optical rain sensor solutions.

[0025] The optical rain sensor solution uses a light rain sensor based on the principle of optical refraction as its core, determining rain intensity by detecting the attenuation of light intensity as raindrops pass through the optical channel. Its core drawback lies in its extremely poor resistance to environmental interference. In scenarios such as direct sunlight, driving against the light, and being illuminated by oncoming headlights at night, light interference can cause the false recognition rate to soar to over 30%. Furthermore, the sensor's installation location (mostly the inside of the windshield) is easily affected by glass stains, fog, and film reflections, and it cannot distinguish between raindrops and debris such as dust and fallen leaves, resulting in a false trigger rate as high as 45% in dusty urban areas. In addition, its sensitivity to light rainfall (0~2.5 mm / h) is insufficient, and its saturation response lag time for heavy rain (>50 mm / h) exceeds 500 ms, making it difficult to meet real-time control requirements.

[0026] The second category is single-vision (camera) recognition solutions.

[0027] This solution relies on road images captured by an onboard forward-facing camera. It uses image processing algorithms (such as edge detection, rain streak trajectory extraction, and deep learning semantic segmentation) to identify rain streak features and determine rain intensity. Its advantage lies in utilizing existing ADAS camera hardware without requiring additional sensors. However, it suffers from inherent technical bottlenecks: in moderate to heavy rain scenarios, raindrops obscuring the lens cause a sharp drop in the image signal-to-noise ratio, rendering rain streak feature extraction ineffective; in low-light nighttime scenarios, reflections from streetlights and vehicle headlights superimposed on the rain streak can lead to misjudgments of rain intensity (e.g., misclassifying light rain as moderate rain); furthermore, the solution is limited by image frame rate (typically 30fps) and algorithm complexity, resulting in end-to-end processing latency generally between 150 and 200 milliseconds, and it consumes significant computational resources, making it prone to stuttering on entry-level ADAS domain control platforms.

[0028] Based on this, embodiments of this disclosure provide a rainfall recognition method that acquires multimodal features related to rainfall through vehicle-mounted sensors and fuses these features to obtain rainfall levels. Multimodality refers to any method or data type capable of sensing, collecting, or characterizing different types of information related to the same physical phenomenon (such as rain). For example, it may include, but is not limited to: acoustic modalities acquired through audio sensors, visual modalities acquired through image sensors, and point cloud modalities acquired through radar sensors. In specific embodiments of this application, multimodality specifically refers to the simultaneous acquisition of audio signals, image signals, and radar point cloud signals from a vehicle-mounted scenario to comprehensively and complementaryly describe the rainfall state. This method can be executed by an vehicle domain controller or ADAS main control unit, or it can be applied to other terminals capable of communicating with the vehicle and / or sensors installed on the vehicle.

[0029] Figure 1The diagram shows the architecture of a rainfall identification system provided in this embodiment. The system includes a sensor module, a data preprocessing module, a feature extraction module, a multimodal fusion module, a rainfall classification module, and an output control module.

[0030] Figure 2 The diagram shows the distribution of the sensor module in the vehicle. The sensor module includes an audio sensor, an onboard radar sensor (such as a 4D millimeter-wave radar), and an onboard camera. The sensors are distributed throughout the vehicle and transmit the collected data to the domain controller through standardized onboard Ethernet, CAN-FD (Controller Area Network Flexible Data Rate), or SERDES (Serial Deserializer) communication interfaces.

[0031] For example, the audio sensor includes an in-vehicle microphone and an external microphone. The collaborative acquisition of audio signals by the in-vehicle and external microphones can construct a "noise reference-target signal enhancement" acoustic processing system. Using ambient noise (engine, air conditioning, etc.) collected by the in-vehicle microphone as a reference signal, adaptive noise cancellation of the external raindrop sound signal is achieved, significantly improving the effectiveness of the audio characteristics. The in-vehicle microphone is an omnidirectional condenser microphone, with microphones arranged in the four sound zones (front, rear, left, and right) inside the vehicle. The audio signals collected by the microphones are transmitted to the domain controller via the vehicle audio bus (A2B). For example, the external microphone is a waterproof and dustproof (MEMS) microphone, which can be installed on the front bumper, rear tailgate, and left and right exterior rearview mirrors. Its sampling rate is consistent with that of the in-vehicle microphone. It is daisy-chained with the in-vehicle microphone via the vehicle audio bus, and together with the data from the in-vehicle microphone, it sends the collected audio signals to the in-vehicle domain controller. The microphones inside and outside the vehicle are used to collect rain audio signals and can also be used to collect voice signals.

[0032] For example, the 4D millimeter-wave radar has stable detection capabilities in all weather and at all times. It adopts the 77GHz frequency band, with the forward radar placed in the center of the grille, and two forward and two rear corner radars placed in the four corners of the vehicle body. The electromagnetic waves can penetrate rain and fog and are not affected by lighting conditions. It can stably acquire point cloud data including the three-dimensional spatial coordinates, radial velocity and reflection intensity of raindrops, and send the data to the domain controller through CAN-FD communication.

[0033] For example, the vehicle-mounted camera uses a forward-facing camera and a 360-degree surround-view camera to capture the visual morphology and spatial distribution density of rain streaks. This allows existing vehicle-mounted cameras to transmit data to the domain controller via SERDES. In well-lit conditions, the vehicle-mounted camera can accurately capture the visual morphology and spatial distribution density of rain streaks, compensating for the lack of detail in radar during light rain.

[0034] The data preprocessing module adopts a "segmented processing + centralized synchronization" architecture, including an audio preprocessing submodule, a radar preprocessing submodule, an image preprocessing submodule, and a data synchronization submodule. Raw data collected by the sensors is first sent to the data preprocessing module for segmented processing. The audio preprocessing submodule preprocesses audio signals (e.g., in-vehicle single-channel audio and external dual-channel audio), the image preprocessing submodule preprocesses video images, and the radar preprocessing submodule preprocesses the collected point cloud data. The data synchronization submodule uses the timestamp of the 4D millimeter-wave radar as a reference and performs time alignment of the audio and image data using alignment methods such as linear interpolation.

[0035] After time alignment, the data enters the feature extraction module, which adopts a "modality-specific extraction + feature standardization" architecture, including an audio feature extraction submodule, a radar feature extraction submodule, an image feature extraction submodule, and a feature normalization submodule. The audio feature extraction submodule extracts rain audio features from the preprocessed audio signal; the radar feature extraction submodule extracts rain point cloud features from the preprocessed point cloud data; the image feature extraction submodule extracts rain image features, including raindrop and raindrop trajectory features; and the feature normalization submodule normalizes the audio, radar, and image feature vectors respectively to ensure consistent feature scale across modalities.

[0036] The multimodal fusion module adopts a two-level fusion architecture of "feature-level fusion + decision-level fusion," including a feature fusion submodule, a decision fusion submodule, and a weight adaptive adjustment submodule. The feature fusion submodule employs an attention-weighted fusion strategy. First, it calculates the weight coefficients of each modal feature through a self-attention mechanism. Then, through concatenation and dimensional mapping of the features from each modality, it unifies the audio, radar, and image features onto a low-dimensional semantic control, obtaining a fused feature vector that integrates all modal features. This fused feature vector serves as the input to the decision fusion submodule. The weight coefficients are calculated based on the Pearson correlation coefficient between each modal feature and the rainfall label; higher correlation results in greater weight and a greater contribution to the fused feature vector.

[0037] The decision fusion submodule is constructed using a Bi-Gated Recurrent Unit (Bi-GRU), whose core lies in the joint modeling of temporal information. The Bi-GRU simultaneously captures the rainfall patterns of the current frame and its preceding and following frames (e.g., the last 5 frames), giving the model a "memory" capability. This means that even if a sensor (e.g., an image) misjudges a "heavy rain" at a certain moment due to strong light reflection, if the radar and audio continuously output "moderate rain" signals, the Bi-GRU will automatically suppress outliers through the synergistic effect of its forward and backward hidden states, outputting a more stable and physically consistent rainfall level confidence vector. This cross-modal, cross-temporal joint reasoning mechanism is a deep manifestation of "fusion" at the decision-making level.

[0038] The adaptive weighting adjustment submodule introduces environmental adaptive factors (light intensity factor, driving speed factor, and noise intensity factor). The light intensity factor is calculated from the average brightness of the images captured by the camera, the driving speed factor is calculated from the real-time vehicle speed obtained from the vehicle's CAN bus, and the noise intensity factor is calculated from the signal-to-noise ratio after audio preprocessing. The submodule dynamically adjusts the weight coefficients of each modality feature during feature fusion through a multi-factor weighting model. This adaptive weighting mechanism, driven by environmental perception factors, is achieved rather than relying on the default adaptive strategies of general large models (such as purely data-driven learning of attention weights). It combines prior knowledge of the physical scene with real-time environmental signals to achieve more controllable and interpretable modality weight adjustment.

[0039] The output control module converts the rainfall level into a signal that can be recognized by the vehicle network and outputs it to the wiper control unit, the lighting control unit, or the main control unit of the advanced driver assistance system through multiple interfaces, while monitoring the sensor's working status and the stability of the recognition results.

[0040] The rainfall classification module adopts a "hard decision + soft correction" mechanism, including a classification submodule and a confidence correction submodule. The classification submodule presets 5 rainfall levels and corresponding judgment thresholds, specifically: no rain, light rain, moderate rain, heavy rain, and rainstorm. The confidence correction submodule uses Bayesian estimation to correct the initial confidence vector output by Bi-GRU, and combines the temporal correlation of historical recognition results (exponential moving average of the recognition results of the last 5 frames) to output the corrected confidence vector, and selects the level with the highest confidence as the current rainfall level.

[0041] The output control module adopts a "multi-interface adaptation + status feedback" architecture, including a data conversion submodule, an interface adaptation submodule, and a status monitoring submodule. The data conversion submodule converts the rainfall level results (digital encoding: 0=no rain, 1=light rain, 2=moderate rain, 3=heavy rain, 4=torrential rain) into CAN bus signals that can be recognized by the vehicle ECU. The interface adaptation submodule provides multiple types of output interfaces, including a CAN FD interface (connecting to the wiper ECU and light ECU), an Ethernet interface (connecting to the ADAS main control unit), and an RS232 interface (for debugging). The status monitoring submodule monitors the working status of each sensor, data transmission quality, and recognition result stability in real time. When a sensor failure occurs (data loss rate > 5%) or the recognition confidence is lower than the threshold (< 0.6), an alarm signal is output to the vehicle instrument panel, and the system switches to a degraded working mode (using historical data for smooth prediction).

[0042] Based on the above system architecture, the rainfall identification method provided in the embodiments of this disclosure will be described in detail below. Figure 3The diagram shown is a flowchart of a rainfall identification method provided in this embodiment of the present disclosure. The method mainly includes the following steps: Step 301: Acquire audio signals collected by the vehicle-mounted audio sensor, images collected by the vehicle-mounted camera, and point cloud data collected by the vehicle-mounted radar sensor.

[0043] For example, the data preprocessing module preprocesses each piece of raw data separately. Specifically, audio signals are preprocessed, including filtering, noise reduction, and amplification; point cloud data is preprocessed, including background filtering and target enhancement; and images are preprocessed, including distortion correction, white balance, and histogram equalization. These preprocessing steps provide high-quality input data for subsequent feature extraction.

[0044] The original audio signals include datasets reflecting the dynamic characteristics of rainfall intensity, collected by in-vehicle and out-of-vehicle microphones. Noise reduction of the audio signals involves using the in-vehicle audio signal collected by the in-vehicle microphone as a reference signal, and denoising the out-of-vehicle audio signal collected by the out-of-vehicle microphone. For example, this involves analyzing the audio characteristics of the reference signal, generating a cancellation signal with the same amplitude but opposite phase as the reference signal, and then superimposing the cancellation signal onto the out-of-vehicle audio signal to obtain the denoised audio signal.

[0045] For example, point cloud data refers to a dataset reflecting the spatial location and motion characteristics of raindrops, acquired by vehicle-mounted radar sensors. Specifically, it can be a set of four-dimensional vectors (x, y, z, v), representing spatial coordinates and radial velocity, respectively, and may also include information such as reflection intensity. Images refer to datasets reflecting the visual morphology and spatial distribution density of rain streaks, collected by vehicle-mounted cameras.

[0046] Step 302: Extract rain audio features from the audio signal, extract rain image features from the image, and extract rain point cloud features from the point cloud data.

[0047] For example, rain audio features refer to features extracted from preprocessed audio signals that characterize the sound state of rainfall or the pattern of raindrop impact. Rain audio features include, but are not limited to, spectral flux, zero-crossing rate, and Mel-frequency cepstral coefficients (MFCC), which directly reflect the frequency and intensity of raindrop impact. As a specific implementation method, a model based on a deep convolutional neural network can be used to automatically learn high-dimensional features from the waveform of the preprocessed audio signal as audio features.

[0048] For example, rain image features refer to features extracted from preprocessed images that are related to the visual appearance of rain streaks or raindrops. Extracting rain image features from preprocessed images includes obtaining features such as rain streak density and raindrop tail length through edge detection, rain streak trajectory extraction, etc. As a specific implementation, a convolutional neural network (such as ResNet) pre-trained on a large number of rainfall images is used to directly output feature maps or feature vectors as image features.

[0049] For example, rain cloud features refer to the features extracted from point cloud data that reflect the spatial distribution and kinematics of raindrops. Rain cloud features include: point cloud density, average radial velocity of raindrops, and mean reflection intensity. Point cloud density is positively correlated with the number of raindrops, and reflection intensity reflects the size of the raindrops. As a specific implementation method, point cloud feature extraction can use a PointNet model or its variants to process unordered point cloud data and generate fixed-dimensional feature vectors. The PointNet model extracts point cloud features by introducing symmetric functions; the specific extraction process is not the focus of this case and will not be detailed here.

[0050] For example, the feature normalization submodule performs scale normalization on modal features such as rain audio features, rain image features, and rain point cloud features to ensure the balance of subsequent fusion.

[0051] Step 303: Combine the rain audio features, the rain image features, and the rain point cloud features to determine the rainfall level; wherein the rainfall level is used to reflect the amount of rainfall.

[0052] For example, there are multiple preset rainfall levels, such as five preset rainfall levels: no rain, light rain, moderate rain, heavy rain, and rainstorm. Each rainfall level can be assigned a corresponding level label.

[0053] In some embodiments, the rain audio features, rain image features, and rain point cloud features each include their respective acquisition timestamps. To ensure the accuracy of data processing, these three modal features are time-aligned before determining the rainfall level. The process of fusing the rain audio features, rain image features, and rain point cloud features to determine the rainfall level includes: time-aligning the rain audio features, rain image features, and rain point cloud features according to their acquisition timestamps; weighting and concatenating the rain audio features, rain image features, and rain point cloud features based on the aligned timestamps to obtain a fused feature vector; constructing a temporal sequence of the fused feature vectors based on the fused feature vectors corresponding to their respective aligned timestamps; and determining the rainfall level based on the temporal sequence.

[0054] For example, using the acquisition timestamp of rain point cloud features as a benchmark, the timestamps of audio data and images are calibrated using linear interpolation, ensuring that each modal feature corresponds to the same rainfall time. This alignment eliminates time deviations caused by different acquisition triggering or transmission delays of multiple sensors, guaranteeing the real-time accuracy of fusion. It is understood that the time alignment operation can be performed directly on the raw data of the modalities such as preprocessed audio signals, images, and point cloud data, or it can align the extracted rain audio features, rain image features, and rain point cloud features; no limitation is made here. Based on the aligned timestamps, the rain audio features, rain image features, and rain point cloud features are weighted, which can be done by multiplying each element of the feature dimension by the corresponding weight coefficient. Through weighting, the contribution of the modal data most relevant to the current rainfall pattern can be highlighted, while the contribution of modal data with low accuracy can be suppressed. A temporal sequence of fused feature vectors is constructed by arranging the fused feature vectors of several consecutive frames in order of their acquisition timestamps, forming a sequence. For example, the fusion feature vectors of five consecutive frames can be selected and arranged in timestamp order to form a time-series sequence for the next decision fusion step. Alternatively, the fusion feature vectors of all parsed frames can be arranged in timestamp order to form a time-series sequence. The time-series sequence captures the dynamic evolution pattern of the rainfall process, providing contextual information for subsequent models.

[0055] In some embodiments, to further enhance the ability to capture the temporal continuity and physical laws of rainfall processes and suppress output fluctuations caused by occasional sensor anomalies, a bidirectional gated cyclic unit (Bi-GRU) is used to construct a decision model. Specifically, determining the rainfall level based on the time series includes: inputting the time series into the decision model to obtain a sequence of confidence values ​​for each candidate rainfall level output by the decision model; wherein the decision model is constructed using a bidirectional gated cyclic unit; and determining the final rainfall level based on the sequence of confidence values ​​for each candidate rainfall level. Inputting the time series into the decision model can be achieved by constructing a sliding window of length M, extracting a subsequence of length M frames from the time series each time, and inputting the subsequence into the decision model. This subsequence includes the current frame for which the rainfall level needs to be determined, as well as multi-mode feature data of the S-frames preceding the current frame and the T-frames following the current frame.

[0056] To improve the reliability of the output confidence values, the structure and training strategy of the decision model were systematically improved. To ensure the accuracy, calibration, and stability of the output confidence, several key optimizations were implemented in the model structure and training process, including: replacing traditional CNNs or unidirectional RNNs with Bi-GRUs to enhance the modeling ability for the temporal dependence of rainfall events; introducing random dropout regularization (dropout rate of 0.3) before the output layer of the decision model to effectively prevent overfitting and improve generalization performance; and using the Softmax activation function in the output layer to ensure that the confidence vector satisfies the probability distribution constraint (the sum of each component is 1, and the value is ∈ [0,1]). The training strategy was optimized as follows: during the model training phase, a multi-task loss function was used for joint optimization, with the main task being the five-class cross-entropy loss and the auxiliary task being the confidence calibration loss, aiming to ensure that the model output confidence is consistent with the actual prediction accuracy. To further mitigate potential overconfidence or underconfidence issues in the model, a temperature scaling technique is introduced during the inference phase. The original unnormalized predicted values ​​(logits) output by the model are sequentially normalized by temperature and transformed using Softmax probability to obtain a calibrated confidence vector. Simultaneously, the training dataset is constructed based on multimodal rainfall data collected from real roads, and data augmentation strategies (such as simulating raindrop noise, image blurring, and radar point cloud sparsification) are employed to improve the model's robustness under complex rainfall conditions. Specifically, the training data for the decision model includes rainfall level labels and the correspondence between fused feature vectors. The output of the decision model is the confidence value of the predicted rainfall level corresponding to each modal feature in the input time series.

[0057] For example, the temporal sequence with fused feature vectors is input into a Bi-GRU model, which encodes the sequence simultaneously in both forward and backward directions. For instance, the Bi-GRU model may include, but is not limited to, a network structure consisting of a forward-gated recurrent unit (GRU) layer and a backward-gated GRU layer. Its output is determined by both the forward and backward hidden states. The forward hidden state captures the rainfall pattern of the previous S frames, while the backward hidden state predicts the rainfall pattern of the current frame lagging behind by T frames, where S and T are integers greater than 1. Together, they provide richer contextual information for the current moment. After the output of the Bi-GRU model is processed by a fully connected layer and activated by Softmax, a sequence of confidence values ​​for each candidate rainfall level (e.g., no rain, light rain, moderate rain, heavy rain, and torrential rain) is obtained.

[0058] In some embodiments, when determining the final rainfall level based on the confidence value sequence output by the Bi-GRU, a correction mechanism is provided to further eliminate abnormal spikes in confidence levels caused by occasional sensor misjudgments or transient environmental interference. Specifically, determining the final rainfall level based on the confidence value sequence of each candidate rainfall level includes: identifying outliers in the confidence value sequence of each candidate rainfall level; correcting the outlier based on the N adjacent confidence values ​​in the confidence value sequence; where N is an integer greater than or equal to 2; and determining the final rainfall level based on the corrected confidence value sequence.

[0059] For example, an outlier can be defined as a confidence value that deviates from the mean of the N adjacent confidence values ​​by more than a certain threshold (e.g., 3 standard deviations), or a confidence value where the difference between confidence values ​​of adjacent frames is greater than a difference threshold (i.e., a jump in confidence value). After identifying an outlier, it is corrected based on the N adjacent confidence values ​​in the sequence. One implementation is to use interpolation or weighted smoothing with the confidence values ​​of the N adjacent frames, for example, replacing the outlier with the mean of the two frames before and after it. The corrected confidence sequence is smoother, avoiding false alarms caused by instantaneous spikes.

[0060] It should be noted that outlier correction can be performed based on a single-modal confidence value sequence. For example, outlier identification can be based on a confidence value sequence of rain audio features, and the outlier can be corrected based on the N consecutive confidence values ​​preceding and following it in the same sequence. Similarly, outlier identification can be based on a confidence value sequence of rain image features, and the outlier can be corrected based on the N consecutive confidence values ​​preceding and following it in the same sequence.

[0061] Specifically, a Bayesian estimation framework can be used for time-series correction. The correction formula is: Pcorrected(yt) = α·Pmodel(yt) + (1-α)·EMA(yt-1,...,yt-5), where Pcorrected(yt) represents the final confidence value sequence after correction at time t, for example, the smoothed output result of a 5-frame confidence value sequence; Pmodel(yt) represents the original confidence value obtained after the output of the Bi-GRU model at time t is processed by a fully connected layer and Softmax activation; EMA(yt-1,...,yt-5) represents the exponential moving average of the corrected confidence value sequence of the most recent 5 consecutive historical frames before the current time, representing the statistical prior information of the latest 5 historical rainfall frames before the current time (i.e., the historical prior probability term of the Bayesian estimation framework); α is the smoothing weighting coefficient, 0<α<1, and can be selected between 0.6 and 0.8. This correction combines the current frame model confidence and historical prior information, which can effectively smooth outliers and output continuous and stable rainfall levels. Finally, the final rainfall level is determined based on the corrected confidence value sequence, for example, selecting the level with the highest confidence. This post-processing further enhances the reliability of the decision output, avoids problems such as wiper malfunctions caused by transient flicker, and improves the safety of downstream control.

[0062] For example, if a frame of an image is misread as a rainstorm due to strong light reflection, and the radar and audio signals in adjacent frames continue to output moderate rain signals, the Bi-GRU will smooth this abnormal spike through bidirectional hidden states. This suppresses the confidence level of rainstorm at that moment and improves the confidence level of moderate rain. Ultimately, a stable rainfall level that conforms to physical laws is determined based on the confidence value sequence of the entire time series. This decision-making method effectively utilizes the complementary advantages of radar's all-weather capability, audio dynamic response, and image details, significantly improving recognition stability in dynamic scenes.

[0063] Understandably, the decision-making model is not limited to the Bi-GRU model. Other models with temporal modeling capabilities can also be used, such as one-dimensional convolutional neural networks with temporal attention mechanisms, Transformer encoders, recurrent neural networks (RNNs), long short-term memory networks (LSTMs), etc., as long as they can capture the dependencies between consecutive frames.

[0064] In a specific example, such as Figure 4The diagram illustrates the decision-making process of the decision fusion submodule, which mainly includes: constructing a temporal sequence of fused feature vectors based on single-frame fusion feature vectors; extracting the current frame and its neighboring frames from the temporal sequence of fused feature vectors to form a temporal sequence of 5 frames; performing bidirectional encoding using a Bi-GRU model, with the forward GRU learning historical rainfall information and the backward GRU learning future context constraints; automatically learning the rainfall level confidence value of the current frame by fusing the forward and backward hidden states; suppressing single-modal outliers based on the obtained rainfall level confidence value sequence to correct misjudged signals; and outputting the rainfall level confidence vector of the current frame based on the modified rainfall level confidence value, which includes the confidence value corresponding to each rainfall level.

[0065] In some embodiments, to further improve the accuracy of the fused feature vector under different environmental conditions and address the problem that fixed weights cannot adapt to complex in-vehicle environments, current environmental information is incorporated into the calculation of the weight coefficients. Specifically, the step of weighting and concatenating the rain audio feature, the rain image feature, and the rain point cloud feature to obtain the fused feature vector includes: determining a first weight value corresponding to the rain audio feature, a second weight value corresponding to the rain image feature, and a third weight value corresponding to the rain point cloud feature based on current environmental information; determining a first weighted value obtained by weighting the first weight value with the rain audio feature, a second weighted value obtained by weighting the second weight value with the rain image feature, and a third weighted value obtained by weighting the third weight value with the rain point cloud feature; and concatenating the first weighted value, the second weighted value, and the third weighted value to obtain the fused feature vector. Here, the strategy of dynamically adjusting the weights using current environmental information improves environmental adaptability. As a specific implementation, the current environmental information can be provided in real time by in-vehicle sensors.

[0066] In some embodiments, the dynamic weight adjustment driven by the environmental adaptive factor does not rely on the default adaptive strategy of the general large model, but combines prior knowledge of the physical scene with real-time environmental signals to achieve controllable and interpretable modal weight adjustment. Specifically, determining the first weight value corresponding to the rain audio feature, the second weight value corresponding to the rain image feature, and the third weight value corresponding to the rain point cloud feature based on the current environmental information includes: determining the environmental adaptive factor based on the current environmental information; wherein the environmental adaptive factor includes at least one of the light intensity factor, driving speed factor, and noise intensity factor; and determining the first weight value corresponding to the rain audio feature, the second weight value corresponding to the rain image feature, and the third weight value corresponding to the rain point cloud feature according to the environmental adaptive factor.

[0067] For example, the illumination intensity factor is calculated based on the average brightness of the images captured by the camera, the driving speed factor is calculated based on the real-time vehicle speed obtained from the vehicle's CAN bus, and the noise intensity factor is calculated based on the signal-to-noise ratio (SNR) after audio preprocessing. These three factors are directly and strongly correlated with the performance of each sensor: illumination intensity directly affects image quality, driving speed affects image motion blur and wind noise, and noise intensity affects the audio SNR. These three factors allow for the quantitative determination of the impact of the current scene on the reliability of each sensor. For instance, an empirical model can be preset: when the illumination intensity factor is below a certain threshold (e.g., 50 lux), the second weight value decreases linearly or non-linearly as the illumination intensity decreases; when the driving speed factor exceeds 80 km / h, the second weight value also decays accordingly; and when the noise intensity factor is above a certain decibel level, the first weight value decreases. This adjustment method based on explicit physical factors enhances the controllability and reliability of the fusion strategy.

[0068] In some embodiments, the higher the noise intensity represented by the noise intensity factor, the smaller the first weight value and the larger the second weight value and / or the third weight value; the lower the light intensity represented by the light intensity factor, the smaller the second weight value and the larger the first weight value and / or the third weight value; the higher the driving speed represented by the driving speed factor, the smaller the first weight value and the second weight value and the larger the third weight value.

[0069] Here, considering the following factors: high noise interferes with audio signal quality; low light (such as at night or in tunnel scenes) reduces image contrast; high-speed driving leads to image motion blur, noise interference, and a relatively enhanced radar Doppler effect. By adjusting the corresponding weight values ​​under these factors, the system can consistently assign greater decision-making power to the currently best-performing sensor in extreme combination scenarios. For example, in a highway scene with heavy rain and nighttime conditions, low light and high noise may reduce both image and audio weights, while significantly increasing radar weight to ensure recognition accuracy. These adjustments can be continuous, such as weights changing linearly or exponentially with the factor, or they can be tiered and abrupt. Specific parameters can be determined through experimental calibration. For another example, at high speeds, the motion blur of raindrops in the image intensifies, reducing the reliability of visual features. In this case, the system correspondingly reduces the image weight and enhances the contribution of millimeter-wave radar and audio signals. For yet another example, in high-noise environments (such as high-speed windshield wipers or tunnel echoes), audio features are interfered with, so the system reduces the audio modality weight and instead relies on radar and visual information.

[0070] For example, an environmental adaptation factor is input into a multi-factor weighted model, which outputs the normalized weight coefficients of each mode in the current context. Specifically, this multi-factor weighted model can calculate the weight values ​​corresponding to each mode based on preset rules (e.g., continuous functions or hierarchical step functions) and empirically calibrated parameters. The model can be set as needed, and the specific form of the multi-factor weighted model is not limited here.

[0071] In some embodiments, in order to provide an objective statistical basis for weight adjustment and avoid the bias caused by setting initial weights based solely on experience, a Pearson correlation coefficient based on calibration data is introduced to initialize the weights. Specifically, determining the first weight value corresponding to the rain audio feature, the second weight value corresponding to the rain image feature, and the third weight value corresponding to the rain point cloud feature based on the current environmental information includes: acquiring calibration data; wherein the calibration data includes the true value of rainfall level, and the true values ​​of audio feature, image feature, and point cloud feature corresponding to the true value of rainfall level; determining the Pearson correlation coefficients between rainfall level and audio feature, image feature, and point cloud feature respectively based on the calibration data; determining the first initial weight value of the rain audio feature, the second initial weight value of the rain image feature, and the third initial weight value of the rain point cloud feature based on the Pearson correlation coefficients; and correcting the first initial weight value of the rain audio feature, the second initial weight value of the rain image feature, and the third initial weight value of the rain point cloud feature based on the current environmental information to obtain the first weight value corresponding to the rain audio feature, the second weight value corresponding to the rain image feature, and the third weight value corresponding to the rain point cloud feature.

[0072] A higher Pearson correlation coefficient indicates a stronger linear correlation between the corresponding modal feature and the rainfall level, and its initial contribution should be greater. For example, if the correlation coefficient for rain audio features is 0.8, for rain image features it is 0.6, and for rain point cloud features it is 0.9, then the initial weights can be normalized to 0.35, 0.26, and 0.39 (keeping the sum to 1). Then, these initial weight values ​​are corrected based on the current environmental information (i.e., the environmental adaptation factor) to obtain the final first, second, and third weight values. The correction process can be achieved by multiplying the initial weights by the environmental adaptation factor. For example, if the current noise intensity is high, the first initial weight value of the rain audio feature can be multiplied by a decay factor less than 1 before re-normalization. In this way, the final weights reflect both the global statistical importance of each modality and adapt to real-time scene changes, comprehensively improving accuracy and robustness. It is understandable that the Pearson correlation coefficient can be replaced with other correlation metrics, such as the Spearman rank correlation coefficient or mutual information, as long as they reflect the correlation between the feature and rainfall. In addition, the correlation coefficient can be recalculated periodically to adapt to sensor aging or changes in climate.

[0073] For example, calibration data can be obtained by simultaneously acquiring multi-sensor data in a controlled rainfall environment and manually labeling it.

[0074] In a specific example, such as Figure 5 The diagram illustrates the process of obtaining the fused feature vectors. It mainly includes: first, preprocessing the original data (audio, radar, and images), then extracting single-modal features to obtain corresponding feature vectors x1, x2, and x3. Subsequently, using calibrated data (including historical rainfall label Y), the Pearson correlation coefficients r1, r2, and r3 between each modal feature and historical rainfall are calculated and normalized to generate basic attention weights, i.e., initial weight values ​​w1, w2, and w3. Modalities with higher correlation are assigned greater weights. Next, each modal feature is multiplied by its corresponding weight to obtain weighted features F1 = w1 × x1, F2 = w2 × x2, and F3 = w3 × x3. These features are then fused using a concatenation operation to obtain a fused feature vector: F_cat = Concat(F1, F2, F3), where the dimension is the sum of the dimensions of each modal feature vector, i.e., d = d1 + d2 + d3, where d1 is the dimension of x1, d2 is the dimension of x2, and d3 is the dimension of x3. Finally, this concatenated feature vector is input to a fully connected layer for dimension mapping and dimensionality reduction, outputting a unified low-dimensional fused feature vector, which serves as the single-frame input for the subsequent Bi-GRU model.

[0075] In a specific example, such as Figure 6The diagram illustrates the process of adaptively adjusting the weights of each modality and outputting confidence scores. It mainly includes: extracting three types of single-modal features based on the original acquired audio data, radar (point cloud data), and image data: rain audio feature x1, rain point cloud feature x2, and rain image feature x3; calculating the Pearson correlation coefficient based on calibration data (including historical rainfall label Y), and obtaining initial weight values ​​using a self-attention mechanism; calculating the illuminance factor from image brightness using real-time environmental data, obtaining the driving speed factor from vehicle speed acquired via the CAN bus, and calculating the confidence scores from audio data... The noise intensity factor is obtained from the SNR. The three types of factors are input into a multi-factor weighted model to obtain normalized correction coefficients. The initial weight values ​​are then adjusted using the correction coefficients to obtain the final weight coefficients for each modality. Based on the final weight coefficients for each modality, the features of each modality are weighted and processed. Concat features are then concatenated to obtain fully connected features. Dimensionality is reduced through dimensionality mapping to obtain a single-frame fused feature vector, which is then input into the Bi-GRU model. The Bi-GRU model outputs the confidence value of the rainfall level, thereby indirectly regulating the output confidence value through environmental adaptive factors.

[0076] In a specific example, an exemplary process for rainfall identification includes: a data preprocessing process, a process for determining weight correction coefficients, a process for determining initial weight values, and a feature fusion process and a decision fusion process for multimodal data.

[0077] The data preprocessing process includes: collecting raw data sources, with each modality acting as an independent front-end branch for data collection, including real-time vehicle speed data obtained through the normal CAN bus, audio signals collected through microphones, point cloud data collected through radar, and image data collected through cameras; unified data preprocessing, including point cloud data and image correction, cropping, and other processing, as well as audio signal noise reduction, and timestamp alignment of multimodal data; and extracting and normalizing features for each modality. The extracted rain image features include rain line features, the rain point cloud features include radar cross section (RCS) statistical features, and the rain audio features include rain acoustic features.

[0078] In the process of determining the modal weight correction coefficient by adaptive environmental factors, real-time vehicle speed data, audio signals and images are input into the environmental factor calculation module to obtain environmental adaptive factors, including noise intensity factors determined by audio SNR, driving speed factors determined by CAN vehicle speed and illumination intensity factors determined by image brightness. Modal weight correction coefficients are obtained by multi-factor weighting.

[0079] The process of obtaining the initial weight values ​​includes: using a controllable rainfall simulation precipitation test device, simultaneously collecting audio signals, images, and point cloud data, as well as the corresponding standard rainfall true values, at five different rainfall levels; archiving the true values ​​of the graded tests to construct a historical rainfall label database as calibration data; this database includes the binding relationship between the true values ​​of each modality feature (audio feature true value, image feature true value, and point cloud feature true value) and the true values ​​of rainfall levels and timestamps; based on this calibration data, determining the Pearson correlation coefficients between rainfall levels and audio features, image features, and point cloud features, respectively; and determining the initial weight values ​​of each modality through a self-attention mechanism.

[0080] The feature fusion process of multimodal data includes: correcting the initial weight values ​​of each modality based on the modality weight correction coefficient to obtain the weight values ​​corresponding to each modality; weighting the multimodal features respectively; concatenating the weighted modal features; and performing full connection and dimensionality reduction to obtain a single-frame fused feature vector.

[0081] The decision fusion process includes: constructing a sliding window of 5 frames; if a frame is missing within the sliding window, it is copied and padded; if a frame is full, the sliding window is updated and the next 5 frames are selected. The time sequence of the 5-frame fused feature vectors is regularized and input into the Bi-GRU model for bidirectional encoding. The reverse GRU learns the context in reverse, and the forward GRU learns the context in forward. Hidden states are concatenated at each time step to obtain global time-series features. Dropout (0.3) regularization is applied, and the original logits are obtained after passing through a fully connected layer. During training, one-hot encoding is generated using the training dataset and experimental graded rainfall data to train ground truth labels and calculate multi-task losses to train the Bi-GRU model. These multi-task losses include cross-entropy (CE) classification loss and expected calibration error (ECE) confidence calibration loss. The total loss is backpropagated to iteratively update the Bi-GRU model and the weights of the fully connected layers. After training, the model is deployed to the inference link for application. In the inference link, temperature scaling and Softmax normalization are performed based on the original logits to obtain the initial confidence vector. Bayesian + Exponential Moving Average (EMA) confidence correction is used, and outlier correction is performed based on only 5 frames of historical EMA values ​​(prior terms) and the current frame confidence. The hard decision rule of 5 rainfall levels is solidified using graded calibration feature thresholds. Based on this hard decision rule, the optimal confidence value is selected from the corrected confidence value vector as the rainfall level. The output rainfall level is encoded into a CAN message and output through multiple peripherals based on CAN-FD Ethernet or RS232, or for fault monitoring, anomaly alarms, and downgraded output when the confidence is low.

[0082] In summary, the core implementation mechanism of the rainfall recognition method provided in this disclosure lies in: utilizing the complementary characteristics of multi-source vehicle-mounted sensors, and overcoming the performance limitations of a single sensor through a multi-level fusion algorithm to achieve accurate rainfall recognition. Specifically, this includes three core aspects. First, the multi-sensor data complementarity mechanism: different sensors possess different advantages in rainfall perception. 4D millimeter-wave radar has all-weather penetration capabilities, accurately capturing the spatial distribution and motion characteristics of raindrops, unaffected by light or raindrop obstruction; external microphones can collect acoustic signals generated by raindrop impacts, reflecting the dynamic changes in rainfall intensity; cameras can capture the visual morphology of rain streaks, intuitively reflecting the spatial distribution of rainfall intensity; and noise signals collected by in-vehicle microphones can be used as a reference to cancel interference from external audio signals, improving the effectiveness of audio features. Through multi-sensor collaborative acquisition, comprehensive perception of rainfall information is achieved. Second, the feature-level fusion enhancement mechanism addresses the issue of rainfall volume characteristics. Rain audio features, rain point cloud features, and rain image features reflect rainfall volume characteristics from different dimensions. The spectral flux and zero-crossing rate of audio features are directly related to the raindrop impact frequency, the point cloud density of radar features is positively correlated with the number of raindrops, and the rain line density of image features is positively correlated with the rainfall intensity. An attention-weighted fusion strategy is employed to automatically focus on feature dimensions strongly correlated with rainfall volume, suppress interfering features, and enhance the discriminative and representational capabilities of the fused features through feature complementarity. Third, the environmental adaptive decision-making mechanism addresses the issue of performance fluctuations in different sensors due to factors such as light intensity and vehicle speed. For example, camera image quality degrades under strong light, and audio noise interference increases at high speeds. An environmental adaptive factor is introduced to dynamically adjust the weight ratio of each modality feature. For instance, the weight of 4D millimeter-wave radar is increased in scenarios such as heavy rain and nighttime, leveraging its strong environmental adaptability to ensure recognition reliability; the weight of image and audio features is appropriately increased in light rain and sunny weather scenarios to improve recognition accuracy. Simultaneously, the Bi-GRU network can capture the temporal variation characteristics of rainfall volume, further enhancing recognition stability in dynamic scenarios.

[0083] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic, and the execution order between steps is not limited to implementation according to step number.

[0084] Figure 7 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0085] This disclosure provides an electronic device comprising: at least one processor 401; at least one memory 402; and one or more I / O interfaces 403 connected between the processor 401 and the memory 402; wherein the memory 402 stores one or more computer programs executable by the at least one processor 401, the one or more computer programs being executed by the at least one processor 401 to enable the at least one processor 401 to perform the above-described rainfall identification method.

[0086] The modules in the aforementioned electronic devices can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0087] This disclosure also provides a computer program product, including a computer program that implements the above-described rainfall identification method when run in a processor.

[0088] The computer program may be stored on a readable storage medium of a computer device or in the cloud; the processor of the computer device reads the computer program from the readable storage medium or in the cloud.

[0089] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically manifested as a computer storage medium; in another optional embodiment, the computer program product is specifically manifested as a software product, such as a software development kit (SDK), etc.

[0090] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0091] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0092] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0093] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0094] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0095] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0096] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0097] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0099] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications or equivalent substitutions made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for identifying rainfall, characterized in that, include: Acquire audio signals collected by the vehicle's audio sensor, images collected by the vehicle's camera, and point cloud data collected by the vehicle's radar sensor; Rain audio features are extracted from the audio signal, rain image features are extracted from the image, and rain point cloud features are extracted from the point cloud data; The rainfall level is determined by integrating the rain audio features, the rain image features, and the rain point cloud features; wherein the rainfall level is used to reflect the amount of rainfall.

2. The method according to claim 1, characterized in that, The process of fusing the rain audio features, the rain image features, and the rain point cloud features to determine the rainfall level includes: The rain audio features, rain image features, and rain point cloud features are time-aligned according to the acquisition timestamps. Based on the aligned timestamps, the rain audio features, rain image features, and rain point cloud features are weighted and then concatenated to obtain a fused feature vector; Based on the fused feature vectors corresponding to the aligned timestamps, a time series sequence of the fused feature vectors is constructed, and the rainfall level is determined based on the time series sequence.

3. The method according to claim 2, characterized in that, The determination of rainfall level based on the time series includes: The time series is input into the decision model to obtain the confidence value sequence of each candidate rainfall level output by the decision model; wherein, the decision model is constructed using a bidirectional gated cyclic unit. The final rainfall level is determined based on the confidence value sequence of each candidate rainfall level.

4. The method according to claim 2, characterized in that, The weighted concatenation of the rain audio features, rain image features, and rain point cloud features yields a fused feature vector, including: Based on the current environmental information, a first weight value corresponding to the rain audio feature, a second weight value corresponding to the rain image feature, and a third weight value corresponding to the rain point cloud feature are determined. A first weighted value is obtained by weighting the first weight value with the rain audio feature, a second weighted value is obtained by weighting the second weight value with the rain image feature, and a third weighted value is obtained by weighting the third weight value with the rain point cloud feature. The first weighted value, the second weighted value and the third weighted value are concatenated to obtain a fused feature vector.

5. The method according to claim 4, characterized in that, The step of determining the first weight value corresponding to the rain audio feature, the second weight value corresponding to the rain image feature, and the third weight value corresponding to the rain point cloud feature based on the current environmental information includes: An environmental adaptation factor is determined based on current environmental information; wherein, the environmental adaptation factor includes at least one of a light intensity factor, a driving speed factor, and a noise intensity factor; The first weight value corresponding to the rain audio feature, the second weight value corresponding to the rain image feature, and the third weight value corresponding to the rain point cloud feature are determined based on the environmental adaptive factor.

6. The method according to claim 5, characterized in that, The step of determining the first weight value corresponding to the rain audio feature, the second weight value corresponding to the rain image feature, and the third weight value corresponding to the rain point cloud feature based on the current environmental information includes: Acquire calibration data; wherein the calibration data includes true values ​​of rainfall levels, and true values ​​of audio features, image features, and point cloud features corresponding to the true values ​​of rainfall levels; Based on the calibration data, the Pearson correlation coefficients between rainfall level and audio features, image features, and point cloud features were determined. Based on the Pearson correlation coefficient, a first initial weight value for the rain audio feature, a second initial weight value for the rain image feature, and a third initial weight value for the rain point cloud feature are determined. Based on the current environmental information, the first initial weight value of the rain audio feature, the second initial weight value of the rain image feature, and the third initial weight value of the rain point cloud feature are corrected to obtain the first weight value corresponding to the rain audio feature, the second weight value corresponding to the rain image feature, and the third weight value corresponding to the rain point cloud feature.

7. The method according to claim 6, characterized in that, The higher the noise intensity represented by the noise intensity factor, the smaller the first weight value, and the larger the second weight value and / or the third weight value; The lower the light intensity represented by the light intensity factor, the smaller the second weight value, and the larger the first weight value and / or the third weight value; The higher the driving speed represented by the driving speed factor, the smaller the first weight value and the second weight value, and the larger the third weight value.

8. The method according to claim 3, characterized in that, The step of determining the final rainfall level based on the confidence value sequence of each candidate rainfall level includes: Identify outliers in the confidence value sequence of each candidate rainfall level, and correct the outliers based on the N confidence values ​​preceding and following the outliers in the confidence value sequence; where N is an integer greater than or equal to 2. The final rainfall level is determined based on the corrected confidence value sequence.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores at least one computer program that can be executed by the at least one processor, the at least one computer program being executed by the at least one processor to enable the at least one processor to perform the rainfall identification method as described in any one of claims 1-8.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when run in a processor, implements the rainfall identification method according to any one of claims 1-8.

Images