A road water depth estimation method, device, equipment and storage medium

Abstract

This application provides a method, apparatus, device, and storage medium for estimating road water accumulation depth. The method includes: determining the water accumulation area of ​​the road based on multimodal perception data of the road where the vehicle is located; the multimodal perception data includes first road surface point cloud data; determining the actual road surface point cloud data of the water accumulation area based on the first road surface point cloud data; determining the first water accumulation depth of the water accumulation area based on the actual road surface point cloud data and the ideal road surface point cloud data; and determining the second water accumulation depth of the water accumulation area based on the first water accumulation depth and the vehicle's motion state information. This application, based on the accurate identification of water accumulation areas using multimodal perception data, first preliminarily determines the water accumulation depth based on the actual and ideal road surface point clouds of the water accumulation area, and then introduces vehicle motion state information to correct the water accumulation depth. This can effectively reduce errors caused by different environmental conditions and driving disturbances, and improve the estimation accuracy of road water accumulation depth.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, specifically to a method, apparatus, device, and storage medium for estimating road water depth. Background Technology

[0002] With the intensification of global climate change and the increasing frequency of extreme rainfall events, urban road flooding has become a significant hazard to road traffic safety. Vehicles unknowingly driving into deep water areas are highly susceptible to engine stalling, electrical system short circuits, and other malfunctions, seriously affecting driving safety. Currently, the depth of road floodwater mainly relies on drivers' subjective estimation using static reference points such as curbs and wheels. However, this method is greatly affected by visibility conditions, driving experience, and the condition of reference points, resulting in poor accuracy in estimating road floodwater depth. Summary of the Invention

[0003] The main objective of this application is to provide a method, apparatus, device, and storage medium for estimating road water depth, aiming to improve the accuracy of road water depth estimation.

[0004] This application provides a method for estimating road flooding depth, comprising: determining a flooded area on the road based on multimodal perception data of the road where the vehicle is located; the multimodal perception data including first road surface point cloud data; determining actual road surface point cloud data of the flooded area based on the first road surface point cloud data of the road; determining a first flooding depth of the flooded area based on the actual road surface point cloud data and ideal road surface point cloud data of the flooded area; the ideal road surface point cloud data being determined based on road surface point cloud data of the flooded area in a dry state; and determining a second flooding depth of the flooded area based on the first flooding depth and the vehicle's motion state information.

[0005] In one embodiment, the multimodal perception data further includes a road surface image; determining the waterlogged area of ​​the road based on the multimodal perception data of the road where the vehicle is located includes: generating a point cloud projection feature map based on the projection result of the first road surface point cloud data of the road in the target image coordinate system; the target image coordinate system is the image coordinate system corresponding to the road surface image; inputting the target data into the trained target neural network model to obtain a water body segmentation mask; the target data includes the road surface image and the point cloud projection feature map; the water body segmentation mask is used to characterize whether each pixel in the road surface image belongs to the waterlogged area; determining the waterlogged area based on the water body segmentation mask.

[0006] In one embodiment, the target data further includes a water body candidate mask; the water body candidate mask is used to guide the target neural network model to identify whether each pixel in the road surface image belongs to a water accumulation area; before inputting the target data into the trained target neural network model to obtain the water body segmentation mask, the method further includes: filtering out first road surface points related to water bodies from the first road surface point cloud based on the reflection intensity of each first road surface point in the first road surface point cloud; the first road surface point cloud data includes the reflection intensity of each first road surface point; and generating the water body candidate mask based on the projection position of the first road surface points related to water bodies in the target image coordinate system.

[0007] In one embodiment, before inputting the target data into the trained target neural network model to obtain the water segmentation mask, the method further includes: acquiring a training dataset; the training dataset includes multiple training samples; each training sample includes a sample road map image, a sample point cloud projection feature map, and a labeled water segmentation mask; at least some of the training samples are obtained under different environmental conditions; inputting the training samples in the training dataset into the neural network model to be trained to obtain a predicted water segmentation mask; determining the target loss function value of the neural network model to be trained based on the predicted water segmentation mask and the labeled water segmentation mask; and updating the network parameters of the neural network model to be trained based on the target loss function value to train the target neural network model.

[0008] In one embodiment, the target neural network model includes an encoder and a decoder; the step of inputting target data into the trained target neural network model to obtain a water segmentation mask includes: downsampling and extracting features from the target data using the encoder to obtain deep semantic features of the target data; upsampling the deep semantic features using the decoder, and fusing the upsampled features with the features extracted by the encoder to obtain the water segmentation mask.

[0009] In one embodiment, the first road surface point cloud data is acquired by vehicle-mounted radar; determining the actual road surface point cloud data of the waterlogged area based on the first road surface point cloud data includes: correcting the first road surface point cloud data based on the vehicle's motion attitude information and the transformation relationship between the vehicle-mounted inertial measurement unit coordinate system and the vehicle-mounted radar coordinate system to obtain the second road surface point cloud data of the road; the motion attitude information is acquired by vehicle-mounted inertial measurement unit; and determining the actual road surface point cloud data of the waterlogged area based on the second road surface point cloud data.

[0010] In one embodiment, the first road surface point cloud data is acquired by vehicle-mounted radar; the ideal road surface point cloud data is point cloud data in a global geographic coordinate system; determining the actual road surface point cloud data of the waterlogged area based on the first road surface point cloud data of the road includes: converting the first road surface point cloud data based on the vehicle's position information in the global geographic coordinate system and the conversion relationship between the vehicle-mounted radar coordinate system and the global geographic coordinate system to obtain the third road surface point cloud data of the road; and determining the actual road surface point cloud data of the waterlogged area based on the third road surface point cloud data of the road.

[0011] In one embodiment, determining the actual road surface point cloud data of the waterlogged area based on the third road surface point cloud data of the road includes: using an iterative nearest-neighbor algorithm to perform corresponding point matching between the third road surface point cloud data and a reference road surface point cloud map to obtain a first matching result; the corresponding point matching is used to match road surface points at the same position in the target plane under the global geographic coordinate system; the target plane is parallel to the road surface; the reference road surface point cloud map is constructed based on the ideal road surface point cloud data; based on the first matching result, updating the position of each third road surface point in the third road surface point cloud data in the target plane to obtain the fourth road surface point cloud data of the road; and filtering road surface point cloud data related to the waterlogged area from the fourth road surface point cloud data of the road to obtain the actual road surface point cloud data of the waterlogged area.

[0012] In one embodiment, the actual road surface point cloud data includes the positions of multiple actual road surface points in the target plane and a first elevation value in the target direction; the ideal road surface point cloud data includes the second elevation values ​​of multiple ideal road surface points in the target direction; the target direction is perpendicular to the road surface; determining the first water depth of the waterlogged area based on the actual road surface point cloud data and the ideal road surface point cloud data includes: for each actual road surface point, querying a reference road surface point cloud map based on the position of the actual road surface point in the target plane to obtain the second elevation value of the target road surface point; the target road surface point is an ideal road surface point whose position in the target plane matches the position of the actual road surface point; determining the water depth of the actual road surface point based on the difference between the first elevation value and the second elevation value; and determining the first water depth of the waterlogged area based on the water depth of each actual road surface point.

[0013] In one embodiment, before querying a reference road surface point cloud map based on the position of the actual road surface point in the target plane to obtain the second elevation value of the target road surface point, the method further includes: using an iterative nearest-point algorithm to perform same-name point matching on the road surface point cloud data collected multiple times in a dry state to obtain a second matching result; based on the second matching result, obtaining multiple sets of same-name road surface point cloud data; each set of same-name road surface point cloud data includes a set of third elevation values ​​of the same-name road surface points in the target direction; the same-name road surface points are used to represent road surface points located at the same position in the target plane; for the i-th set of same-name road surface points, based on the third elevation values ​​of each same-name road surface point, determining the target elevation value of the i-th set of same-name road surface points in the target direction; the i-th set of same-name road surface points is the same-name road surface point corresponding to any set of same-name road surface point cloud data; constructing the reference road surface point cloud map based on the position of each set of same-name road surface points in the target plane and the target elevation value in the target direction.

[0014] In one embodiment, determining the second water depth of the waterlogged area based on the first water depth and the vehicle's motion state information includes: determining a water depth correction amount based on the vehicle's motion state information; and correcting the first water depth using the water depth correction amount to obtain the second water depth.

[0015] In one embodiment, after determining the second water depth of the waterlogged area based on the first water depth and the vehicle's motion state information, the method further includes: determining the waterlogging risk level of the waterlogged area based on a comparison result between the second water depth and a water depth threshold; and controlling the vehicle to perform corresponding hazard avoidance operations based on the waterlogging risk level.

[0016] This application embodiment also provides a road waterlogging depth estimation device, the device including a waterlogging area determination module, a point cloud data determination module, a first waterlogging depth determination module, and a second waterlogging depth determination module; the waterlogging area determination module is used to determine the waterlogging area of ​​the road based on multimodal perception data of the road where the vehicle is located; the multimodal perception data includes first road surface point cloud data; the point cloud data determination module is used to determine the actual road surface point cloud data of the waterlogging area based on the first road surface point cloud data of the road; the first waterlogging depth determination module is used to determine the first waterlogging depth of the waterlogging area based on the actual road surface point cloud data and ideal road surface point cloud data of the waterlogging area; the ideal road surface point cloud data is determined based on the road surface point cloud data of the waterlogging area in a dry state; the second waterlogging depth determination module is used to determine the second waterlogging depth of the waterlogging area based on the first waterlogging depth and the vehicle's motion state information.

[0017] This application embodiment also provides a road water depth estimation device, the device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described road water depth estimation method.

[0018] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for estimating road water depth.

[0019] This application provides a method, apparatus, device, and storage medium for estimating road water depth. Based on the accurate identification of waterlogged areas using multimodal sensing data, the method first preliminarily determines the water depth based on the actual road surface point cloud and the ideal road surface point cloud of the waterlogged area, and then introduces vehicle motion state information to correct the water depth. This can effectively reduce the errors caused by different environmental conditions and driving disturbances, and improve the estimation accuracy of road water depth. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the road water depth estimation method provided in the embodiments of this application.

[0021] Figure 2 This is a multimodal data schematic diagram of the road water depth estimation method provided in the embodiments of this application.

[0022] Figure 3 This is a schematic diagram illustrating the specific process of the road water depth estimation method provided in the embodiments of this application.

[0023] Figure 4 This is a schematic diagram of the road water depth estimation device provided in the embodiments of this application.

[0024] Figure 5 This is a schematic diagram of the road water depth estimation device provided in the embodiments of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0026] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the digit " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0027] The road water depth estimation method provided in this application can be applied to a road water depth estimation device or its software. The road water depth estimation device can be a vehicle, a terminal, or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers. The software can be an application that implements the road water depth estimation method, but is not limited to the above forms.

[0028] The method for estimating road water depth provided in this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Please see Figure 1 The method for estimating road water depth provided in this application embodiment may include: Step S101: Based on the multimodal perception data of the road where the vehicle is located, determine the water accumulation area of ​​the road; the multimodal perception data includes the first road surface point cloud data; Optionally, the multimodal perception data may also include road surface images. Road surface images can be obtained by capturing images of the road surface using an onboard camera; the first road surface point cloud data can be obtained by scanning the road surface using onboard radar.

[0030] In practical implementation, three-dimensional geometric features of the road surface can be extracted based on the first road surface point cloud data, and suspected water accumulation areas can be identified by combining the reflection intensity and echo width of the point cloud. Simultaneously, color and texture features can be extracted from the road surface image, and high-reflectivity areas (i.e., suspected water accumulation areas) can be obtained through threshold segmentation. Finally, the intersection of the suspected water surface areas determined based on the multimodal sensing data can be taken to obtain the water accumulation areas of the road. Alternatively, target data can be determined first based on multimodal sensing data, and then the target data can be input into the trained target neural network to obtain the water accumulation areas of the road. Specific implementation details can be found in the relevant descriptions below and will not be described here.

[0031] Step S102: Based on the first road surface point cloud data of the road, determine the actual road surface point cloud data of the waterlogged area; In practice, road surface point cloud data belonging to the waterlogged area can be filtered from the first road surface point cloud data to obtain the actual road surface point cloud data of the waterlogged area. Alternatively, the first road surface point cloud data can be corrected using vehicle motion attitude information, and / or transformed based on the real-time position of the vehicle, and then the actual road surface point cloud data of the waterlogged area can be determined based on the corrected and / or transformed road surface point cloud data. For specific implementation details, please refer to the relevant descriptions below, which will not be described here.

[0032] Step S103: Determine the first water depth of the waterlogged area based on the actual road surface point cloud data and the ideal road surface point cloud data; the ideal road surface point cloud data is determined based on the road surface point cloud data of the waterlogged area in a dry state. Optionally, the actual road surface point cloud data of the waterlogged area may include the position of the water surface points in the waterlogged area within the road surface and their elevation values ​​perpendicular to the road surface; the ideal road surface point cloud data of the waterlogged area may include the position of the road surface points (which may be called ideal road surface points) in the waterlogged area under dry conditions within the road surface and their elevation values ​​perpendicular to the road surface.

[0033] In practice, a first average elevation value for each water surface point in the flooded area and a second average elevation value for each ideal road surface point can be calculated. Then, the first water depth of the flooded area can be determined based on the difference between the first and second averages. Alternatively, for each water surface point in the flooded area, a target road surface point located at the same position within the road surface can be selected from the ideal road surface points of the flooded area. Then, the water depth of that water surface point can be determined based on the difference between its elevation and the target road surface point's elevation. Finally, the first water depth of the flooded area can be determined based on the water depth of each water surface point.

[0034] Step S104: Based on the first water depth and the vehicle's motion status information, determine the second water depth of the water accumulation area.

[0035] Optionally, the vehicle's motion status information may include at least one of pitch angle, roll angle, and speed.

[0036] In practice, the first water depth and the vehicle's motion status information can be used as a joint query index to search the pre-generated motion status-water depth mapping table and directly match to obtain the second water depth. Alternatively, the vehicle's motion status information can be used to determine the water depth correction amount; then, the water depth correction path can be used to correct the first water depth to obtain the second water depth. For specific implementation details, please refer to the relevant descriptions below, which will not be described here.

[0037] Based on the accurate identification of waterlogged areas using multimodal perception data, this embodiment first preliminarily determines the water depth based on the actual road surface point cloud and the ideal road surface point cloud of the waterlogged area, and then introduces vehicle motion state information to correct the water depth. This can effectively reduce the errors caused by different environmental conditions and driving disturbances, and improve the estimation accuracy of road water depth.

[0038] Optionally, the multimodal perception data may also include road surface images. In one embodiment, determining the waterlogged area of ​​the road based on the multimodal perception data of the road where the vehicle is located in step S101 above includes: A point cloud projection feature map is generated based on the projection result of the first road surface point cloud data of the road in the target image coordinate system; the target image coordinate system is the image coordinate system corresponding to the road surface image. The target data is input into the trained target neural network model to obtain the water body segmentation mask; the target data includes road surface images and point cloud projection feature maps; the water body segmentation mask is used to characterize whether each pixel in the road surface image belongs to a water accumulation area. The water accumulation area is determined based on the water body segmentation mask.

[0039] In practical implementation, the reflection intensity and elevation values ​​of each point cloud in the first-path point cloud data can be projected onto the target image coordinate system using the pre-calibrated intrinsic parameter matrix of the vehicle-mounted camera and the extrinsic parameter matrix between the vehicle-mounted camera and the vehicle-mounted radar, thus obtaining a point cloud projection feature map. Optionally, the intrinsic parameter matrix, used to characterize the transformation relationship between the vehicle-mounted camera coordinate system and the target image coordinate system, can be obtained using the Zhang Zhengyou calibration method; the extrinsic parameter matrix, used to characterize the transformation relationship between the vehicle-mounted camera coordinate system and the vehicle-mounted radar coordinate system, can be obtained by using the least squares method iterative optimization to minimize the projection error between the inner corner points of the first-path point cloud and the calibration board image.

[0040] Optionally, if multiple first road surface points are projected onto the same pixel in the target image coordinate system, the mean of the reflection intensity and the minimum of the elevation value of the multiple first road surface points can be taken as the feature value of the pixel. In this way, the sparse road surface point cloud can be densed into a two-channel point cloud projection feature map, where the first channel can represent the reflection intensity and the second channel can represent the elevation value.

[0041] Optionally, the target neural network model mentioned above is a neural network model that has at least learned the mapping relationship between the road surface image, the point cloud projection feature map and the water body segmentation mask. In actual implementation, the road surface image and the point cloud projection feature map can be input into the target neural network model to obtain the water body segmentation mask.

[0042] Optionally, the water body segmentation mask can be represented as a set of mask data corresponding to the road surface image. For example, if the water body segmentation mask of pixel A in the road surface image is 1, it means that pixel A belongs to a waterlogged area; if the water body segmentation mask of pixel B is 0, it means that pixel B does not belong to a waterlogged area. The water body segmentation mask can also be represented as a binary image of the same size as the road surface image. The pixel values ​​in the binary image consist of 0 and 1, where 1 indicates that the pixel belongs to a waterlogged area, and 0 indicates that the pixel does not belong to a waterlogged area.

[0043] In practice, connected component analysis can be performed on the water body segmentation mask to obtain the pixel set of each water accumulation area; then the road surface area corresponding to each pixel set can be determined as the water accumulation area.

[0044] This application embodiment generates a point cloud projection feature map by projecting the first road surface point cloud data onto the target image coordinate system, and uses the road surface image and the point cloud projection feature map as model inputs. The trained target neural network outputs a water body segmentation mask. On the one hand, it can effectively overcome the problem that single-modal perception data is easily interfered with by environmental conditions such as illumination, reflection, and occlusion, and achieve accurate discrimination of whether each pixel in the road surface image belongs to a water accumulation area. On the other hand, it utilizes the advantages of the target neural network model in adaptive fusion and learning of multimodal features, thereby improving the efficiency and accuracy of identifying road water accumulation areas.

[0045] Optionally, the target data also includes a water body candidate mask; the water body candidate mask can be used to guide the target neural network model to identify whether each pixel in the road surface image belongs to a waterlogged area. In one embodiment, before inputting the target data into the trained target neural network model to obtain the water body segmentation mask, the road waterlogging depth estimation method provided in this application embodiment further includes: Based on the reflection intensity of each first-path surface point in the first-path surface point cloud, first-path surface points related to water bodies are selected from the first-path surface point cloud; the first-path surface point cloud data includes the reflection intensity of each first-path surface point. Based on the projection position of the first path point related to the water body in the target image coordinate system, a water body candidate mask is generated.

[0046] In practical implementation, considering the low reflectivity of vehicle-mounted radar to water, the reflection intensity of each first-path surface point can be compared with a reflection intensity threshold to obtain a set of first-path surface points with reflection intensities lower than the threshold. The first-path surface points within this set can then be identified as those associated with the water body. Optionally, the reflection intensity threshold can be obtained by performing binary classification statistics on the reflection intensities of surface points in water-filled and non-water-filled areas collected by the vehicle-mounted radar using the Otsu's method.

[0047] Furthermore, for each first path point related to the water body, the projection position of the first path point in the target image coordinate system can be determined using the aforementioned intrinsic and extrinsic parameter matrices. The pixels at these projection positions can be labeled as belonging to the water body region, thus obtaining a water body candidate mask corresponding to the path image. Optionally, the representation of the water body candidate mask can refer to the representation of the water body segmentation mask described above, and will not be repeated here.

[0048] This application embodiment filters out water-related first-path points from the first-path point cloud based on the reflection intensity of each first-path point; and generates a water body candidate mask based on the projection position of the water-related first-path points in the target image coordinate system, which can improve the accuracy of the water body candidate mask. On this basis, the road surface image, point cloud projection feature map, and water body candidate mask are input together into the trained target neural network model. Under the guidance of the water body candidate mask, the target neural network model can output a more accurate water body segmentation mask based on the road surface image and point cloud projection feature map, thereby further improving the accuracy of identifying road waterlogged areas.

[0049] In one embodiment, before inputting the target data into the trained target neural network model to obtain the water body segmentation mask, the road water depth estimation method provided in this application embodiment further includes: Obtain the training dataset; the training dataset includes multiple training samples; each training sample includes a sample road surface image, a sample point cloud projection feature map, and a labeled water body segmentation mask; at least some of the training samples were obtained under different environmental conditions. Input the training samples from the training dataset into the neural network model to be trained to obtain the predicted water body segmentation mask; Based on the predicted water segmentation mask and the labeled water segmentation mask, determine the target loss function value of the neural network model to be trained; Based on the target loss function value, the network parameters of the neural network model to be trained are updated to obtain the target neural network model.

[0050] In practice, under different weather and lighting conditions, such as sunny days, rainy days, nighttime, and backlighting, road surface images and point cloud data can be collected using vehicle-mounted cameras and radar for roads with varying water depths (e.g., 0cm-50cm). At least a portion of the collected road surface images and point cloud data can then be divided into training datasets.

[0051] Next, the following steps can be performed on the training samples in the training dataset: The training samples are input into the neural network model to be trained to obtain the predicted water body segmentation mask of the training samples. Based on the predicted water segmentation mask and the labeled water segmentation mask, determine the target loss function value of the neural network model to be trained; If the target loss function value does not meet the training stopping condition, the model parameters of the neural network model to be trained are adjusted to obtain an updated neural network model. The updated neural network model is then trained using the next training sample until the training stopping condition is met, resulting in a trained target neural network model.

[0052] To further improve the robustness of the target neural network model, the collected road surface images and road surface point cloud data can be divided into training datasets, validation datasets, and test datasets according to a preset ratio. Optionally, the preset ratio can be 7:2:1. After each round of training of the network model using the training samples and the untrained data, the performance of the model can be validated using the validation dataset; after obtaining the target neural network model, its performance can be tested using the test dataset. This can improve the robustness of the target neural network model.

[0053] Optionally, during model training, the optimizer can be an Adaptive Moment Estimation (Adam) optimizer, with an initial learning rate of 1e-4, 100 iterations, and a learning rate decay of 1 / 10 every 20 iterations.

[0054] In practical implementation, the cross-entropy loss function can be used to determine the first loss function value based on the predicted water body segmentation mask and the labeled water body segmentation mask; and the dice loss function can be used to determine the second loss function value based on the predicted water body segmentation mask and the labeled water body segmentation mask; then, the target loss function value can be determined based on the first and second loss function values ​​according to preset weights. The cross-entropy loss function is used to measure the error of pixel-level classification; the Dice loss function is used to alleviate the problem of class imbalance between positive and negative samples and improve the segmentation recall rate of water accumulation areas; the preset weights can be 1:1.

[0055] It is worth mentioning that, in the case that the target data includes water body candidate masks, the training samples can also include sample water body candidate masks. The method of obtaining them can refer to the method of obtaining water body candidate masks mentioned above, and will not be repeated here.

[0056] This application embodiment trains a neural network model using training samples obtained under different environmental conditions, enabling the model to fully learn the generalization characteristics of waterlogged areas in diverse scenarios. Furthermore, during training, cross-entropy loss and Dice loss functions are introduced to jointly optimize the model parameters, guiding the model to balance global accuracy and local integrity. The resulting target neural network model possesses stronger generalization and anti-interference capabilities, allowing for accurate identification of waterlogged areas on roads.

[0057] Optionally, the aforementioned target neural network model includes an encoder and a decoder. In one embodiment, the process of inputting target data into the trained target neural network model to obtain a water segmentation mask includes: The target data is downsampled and its features are extracted by the encoder to obtain the deep semantic features of the target data; The deep semantic features are upsampled by the decoder, and the upsampled features are fused with the features extracted by the encoder to obtain the water body segmentation mask.

[0058] Optionally, the target neural network model can be a lightweight water segmentation model based on a simplified U-Net architecture, with ≤5M model parameters and an inference speed ≥20fps based on NVIDIA Jetson Xavier NX edge computing units, balancing segmentation accuracy and inference speed. Its specific structure can be as follows: ① Encoder section: Contains 3 convolutional layers and 2 max pooling layers. The convolutional layers use 3×3 kernels with a stride of 1, same padding, and ReLU activation function. The aligned road map image (which can be a 3-channel RGB image), point cloud projection feature map (2 channels), and water body candidate mask (1 channel) are concatenated into a 6-channel base input to the first convolutional layer, with an output of 32 channels. The second convolutional layer outputs 64 channels, and the third convolutional layer outputs 128 channels. The max pooling layer uses a 2×2 kernel with a stride of 2 to downsample and preserve feature information.

[0059] ② Decoder section: Contains 2 deconvolutional layers and 2 concatenation layers. The deconvolutional layers use 2×2 convolutional kernels with a stride of 2 to achieve feature map upsampling; the first deconvolutional layer maps the 128-channel feature map to 64 channels, concatenates it with the output feature map of the second layer of the encoder, and then optimizes the output of 64-channel features through convolutional layers; the second deconvolutional layer maps the 64-channel feature map to 32 channels, concatenates it with the output feature map of the first layer of the encoder, and then optimizes the output of 32-channel features through convolutional layers; finally, a 1×1 convolutional layer maps the 32-channel features to a single-channel water segmentation mask.

[0060] This application embodiment achieves water body segmentation by using a target neural network model with an encoder-decoder structure. The encoder extracts deep semantic features of the target data through step-by-step downsampling, which can capture the global context information of the water accumulation area. The decoder recovers shallow features through step-by-step upsampling and uses skip connections to achieve the fusion of shallow features and deep semantic features at each level. This can effectively preserve the edge details and spatial location information of the water accumulation area, and improve the accuracy and robustness of identifying road water accumulation areas using the target neural network model.

[0061] Optionally, the first road surface point cloud data is obtained through vehicle-mounted radar. In one embodiment, step S102 above: determining the actual road surface point cloud data of the waterlogged area based on the first road surface point cloud data includes: Based on the vehicle's motion attitude information and the transformation relationship between the vehicle's inertial measurement unit coordinate system and the vehicle's radar coordinate system, the first road surface point cloud data is corrected to obtain the second road surface point cloud data; the motion attitude information is acquired by the vehicle's inertial measurement unit. Based on the second road surface point cloud data, the actual road surface point cloud data of the waterlogged area is determined.

[0062] Considering that the road surface point cloud data collected by vehicle-mounted radar on the same road may differ under different vehicle motion postures, in order to eliminate the influence of vehicle motion posture, in practical implementation, the vehicle motion posture information synchronously collected by the vehicle-mounted inertial measurement unit (IMU) can be used to correct the first road surface point cloud data collected by the vehicle-mounted radar to obtain the second road surface point cloud data. This can correct the point cloud tilt caused by vehicle pitch, roll, and other postures. Then, based on the second road surface point cloud data of the road, the actual road surface point cloud data of the water accumulation area can be determined. This can ensure that the actual road surface point cloud data of the determined water accumulation area has eliminated the influence of vehicle motion posture, thereby improving the accuracy of determining the water depth based on the actual road surface point cloud data of the water accumulation area.

[0063] Optionally, the vehicle's motion attitude information may include at least one of the vehicle's pitch angle and roll angle. In practice, a multi-attitude calibration experiment (vehicle stationary, pitch, roll, etc.) can be used to solve the rotation matrix from the vehicle's inertial measurement unit coordinate system to the vehicle's radar coordinate system, obtaining the transformation relationship between the two. Then, this rotation matrix can be used to convert the first road surface point cloud data into second road surface point cloud data. Then, road surface point cloud data belonging to the waterlogged area can be filtered from the second road surface point cloud data to obtain the actual road surface point cloud data of the waterlogged area. If the ideal road surface point cloud data is point cloud data in a global geographic coordinate system, the second road surface point cloud data can also be converted to the global geographic coordinate system based on the vehicle's real-time position. Then, the actual road surface point cloud data of the waterlogged area can be determined based on the converted road surface point cloud data to unify the coordinate systems of the actual road surface point cloud data and the ideal road surface point cloud data, avoiding deviations in the determined water depth due to coordinate system inconsistencies. Specific implementation details are provided below and will not be described here.

[0064] This application embodiment first uses the vehicle's motion posture information to correct the first road surface point cloud data to obtain the second road surface point cloud data; then, based on the second road surface point cloud data, it determines the actual road surface point cloud data of the water accumulation area. This can effectively eliminate the point cloud tilt caused by the vehicle's motion posture, thereby improving the accuracy of determining the road water depth based on the determined actual road surface point cloud data of the water accumulation area.

[0065] Optionally, the first road surface point cloud data is obtained through vehicle-mounted radar; ideally, the road surface point cloud data can be point cloud data in a global geographic coordinate system. In one embodiment, step S102 above: determining the actual road surface point cloud data of the waterlogged area based on the first road surface point cloud data includes: Based on the vehicle's location information in the global geographic coordinate system and the conversion relationship between the vehicle radar coordinate system and the global geographic coordinate system, the first road table point cloud data is converted to obtain the third road table point cloud data. Based on the third road surface point cloud data of the road, the actual road surface point cloud data of the waterlogged area is determined.

[0066] Considering that the radar installation locations of different vehicles may vary, the initial road surface point cloud data collected by each vehicle's onboard radar for the same road may differ. If the actual road surface point cloud data of the flooded area is directly determined based on this initial data, to ensure the accuracy of subsequent water depth estimation, ideal road surface point cloud data for the flooded area needs to be collected separately for each vehicle. Then, the ideal road surface point cloud data corresponding to that vehicle is retrieved based on the vehicle information to determine the water depth. This would reduce the efficiency of water depth estimation. To avoid this problem, preferably, the ideal road surface point cloud data for the flooded area can be point cloud data in a global geographic coordinate system. This eliminates the need to restrict the vehicles collecting the ideal road surface point cloud data for the flooded area. Optionally, the global geographic coordinate system can be a fixed spatial coordinate system based on the Earth.

[0067] In practice, the vehicle's real-time dynamic differential global positioning system (RTK-GPS) can be used to collect the vehicle's real-time position coordinates, obtaining the vehicle's position information in the global geographic coordinate system. Then, using the pre-calibrated conversion relationship between the vehicle's onboard radar coordinate system and the global geographic coordinate system, the first-path point cloud data can be converted into third-path point cloud data based on the vehicle's position information in the global geographic coordinate system.

[0068] Then, road surface point cloud data belonging to the waterlogged area can be directly filtered from the third road surface point cloud data to obtain the actual road surface point cloud data of the waterlogged area; alternatively, the third road surface point cloud data can be corrected using vehicle motion attitude information, and then the actual road surface point cloud data of the waterlogged area can be determined based on the corrected road surface point cloud data to correct the point cloud tilt caused by vehicle motion attitude; alternatively, the Iterative ClosestPoint (ICP) algorithm can be used to match the third road surface point cloud data with the reference road surface point cloud map, and the road surface point cloud data of the waterlogged area can be determined based on the matching results. For specific implementation details, please refer to the relevant descriptions below, which will not be described here.

[0069] This application embodiment transforms the first road surface point cloud data based on the vehicle's position information in the global geographic coordinate system and the transformation relationship between the vehicle radar coordinate system and the global geographic coordinate system to obtain the third road surface point cloud data of the road; and based on the third road surface point cloud data of the road, determines the actual road surface point cloud data of the waterlogged area, realizing the unification of the actual road surface point cloud data and the ideal road surface point cloud data of the waterlogged area into the global geographic coordinate system, which can effectively eliminate the point cloud misalignment caused by vehicle position deviation and improve the estimation efficiency and accuracy of road waterlog depth.

[0070] In one embodiment, the above-mentioned third road surface point cloud data based on roads is used to determine the actual road surface point cloud data of the waterlogged area, including: Using the iterative nearest point algorithm, the third road surface point cloud data and the reference road surface point cloud map are matched for corresponding points to obtain the first matching result; corresponding point matching is used to match road surface points that are at the same position in the target plane under the global geographic coordinate system; the target plane is parallel to the road surface; the reference road surface point cloud map is constructed based on the ideal road surface point cloud data; Based on the first matching result, update the position of each third table point in the third table point cloud data in the target plane to obtain the fourth table point cloud data of the road; By filtering road surface point cloud data related to the waterlogged area from the fourth road surface point cloud data, the actual road surface point cloud data of the waterlogged area is obtained.

[0071] Optionally, if the distance between any pair of road surface points in the target plane is less than a preset distance or their positions completely overlap, the pair of road surface points can be considered to be at the same position in the target plane.

[0072] To improve the matching efficiency between actual and ideal road surface points in the waterlogged area during the subsequent determination of water depth, the ICP algorithm can be used in practice to determine the ideal road surface points that are located at the same position in the target plane as each of the third road surface points from the reference road surface point cloud map.

[0073] Next, for each third road point, its position in the target plane can be updated to the position of the ideal road point in the target plane that matches it, while keeping its elevation value unchanged. In this way, fourth road point cloud data that is spatially aligned with the ideal road point cloud data in the reference road point cloud map can be obtained.

[0074] Then, based on the location range of the waterlogged area, the corresponding spatial filtering area can be determined. Then, by traversing the fourth road surface point cloud data of the road, the point cloud falling within the spatial filtering area is retained as a valid point cloud, and redundant point clouds outside the area are removed, thus obtaining the actual road surface point cloud data that is only related to the waterlogged area.

[0075] This application embodiment utilizes an iterative nearest-point algorithm to match corresponding points between the third road surface point cloud data and the reference road surface point cloud map, obtaining a first matching result; and based on the first matching result, updates the position of each third road surface point in the third road surface point cloud data within the target plane to obtain the fourth road surface point cloud data of the road; and filters road surface point cloud data related to the waterlogged area from the fourth road surface point cloud data of the road to obtain the actual road surface point cloud data of the waterlogged area. This can improve the efficiency and accuracy of using the actual road surface point cloud of the waterlogged area to query the reference road surface point cloud map to obtain the ideal road surface point that matches it in the subsequent waterlogging depth estimation process, thereby improving the efficiency and accuracy of subsequent road waterlogging depth estimation.

[0076] In one embodiment, the above-mentioned third road surface point cloud data based on roads is used to determine the actual road surface point cloud data of the waterlogged area, including: Based on the vehicle's location information in the global geographic coordinate system and the conversion relationship between the vehicle radar coordinate system and the global geographic coordinate system, the first road table point cloud data is converted to obtain the third road table point cloud data. Based on the vehicle's motion attitude information and the transformation relationship between the vehicle's inertial measurement unit coordinate system and the vehicle's radar coordinate system, the third road meter point cloud data is corrected to obtain the fifth road meter point cloud data. Using the iterative nearest point algorithm, the point cloud data of the fifth road table is matched with the reference road table point cloud map to obtain the target matching result; Based on the target matching results, update the position of each fifth table point in the fifth table point cloud data in the target plane to obtain the sixth table point cloud data of the road; By filtering point cloud data related to the waterlogged area from the sixth road surface point cloud data, the actual road surface point cloud data of the waterlogged area is obtained.

[0077] The specific implementation process of this embodiment can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0078] In this embodiment, the first road surface point cloud data is converted to a global geographic coordinate system to obtain the third road surface point cloud data, which can effectively eliminate the point cloud misalignment caused by vehicle position shift. Then, the third road surface point cloud data is corrected using vehicle motion attitude information to obtain the fifth road surface point cloud data, which can effectively eliminate the point cloud tilt caused by vehicle motion attitude. Finally, the iterative nearest point algorithm is used to update the fifth road surface point cloud data to a sixth road surface point cloud data that matches the corresponding point in the reference road surface point cloud map. The actual road surface point cloud data of the waterlogged area is then selected from the sixth road surface point cloud data. This can improve the efficiency and accuracy of using the actual road surface point cloud of the waterlogged area to query the reference road surface point cloud map to obtain the ideal road surface point that matches it in the subsequent waterlogging depth estimation process, thereby improving the estimation efficiency and accuracy of road waterlogging depth.

[0079] Optionally, the actual road surface point cloud data may include the positions of multiple actual road surface points in the target plane and the first elevation value in the target direction; the ideal road surface point cloud data may include the second elevation values ​​of multiple ideal road surface points in the target direction; and the road surface is perpendicular to the road surface in the target direction.

[0080] In one embodiment, determining the first water depth of the waterlogged area based on the actual road surface point cloud data and the ideal road surface point cloud data in step S103 above includes: For each actual road surface point, based on the actual road surface point's position in the target plane, a reference road surface point cloud map is queried to obtain the second elevation value of the target road surface point; the target road surface point is an ideal road surface point that is located at the same position as the actual road surface point in the target plane. The water depth at the actual road surface point is determined based on the difference between the first elevation value and the second elevation value. The first water depth of the waterlogged area is determined based on the water depth at each actual road surface point.

[0081] Optionally, the reference road surface point cloud map stores ideal road surface point cloud data. The ideal road surface point cloud data may include the position of each ideal road surface point in the target plane and its elevation value in the target direction.

[0082] In practice, for each actual road surface point in the waterlogged area, the elevation value of the ideal road surface point that is at the same position as the actual road surface point in the target plane can be determined by querying the reference road surface point cloud map using the position of the actual road surface point in the target plane, thus obtaining the second elevation value of the target road surface point.

[0083] The difference between the first elevation value and the second elevation value can be directly determined as the first water depth of the waterlogged area; alternatively, a coefficient can be added to the difference between the first elevation value and the second elevation value to obtain the first water depth of the waterlogged area. The specific implementation can be determined according to the actual situation, and no limitation is made here.

[0084] Optionally, the first water depth of the waterlogged area can be represented as a set or as a single value. If it is a set, the first water depth of the waterlogged area can include the water depth of each actual road surface point; if it is a single value, the average water depth of each actual road surface point can be determined as the first water depth of the waterlogged area; alternatively, the maximum water depth of each actual road surface point can be determined as the first water depth of the waterlogged area.

[0085] This application embodiment improves the estimation accuracy of road water depth by querying a reference road point cloud map for each actual road surface point based on its position in the target plane; determining the water depth of the actual road surface point based on the difference between the first and second elevation values; and determining the first water depth of the waterlogged area based on the water depth of each actual road surface point.

[0086] In one embodiment, before obtaining the second elevation value of the target road surface point by querying a reference road surface point cloud map based on the actual road surface point's location in the target plane, the road water accumulation depth estimation method provided in this application embodiment further includes: Using the iterative nearest point algorithm, the road surface point cloud data collected multiple times under dry conditions are matched with corresponding points to obtain a second matching result; Based on the second matching result, multiple sets of road surface point cloud data with the same name are obtained; each set of road surface point cloud data with the same name includes a third elevation value of a set of road surface points in the target direction; road surface points with the same name are used to represent road surface points that are in the same position in the target plane; For the i-th group of road surface points with the same name, the target elevation value of the i-th group of road surface points with the same name is determined in the target direction based on the third elevation value of each road surface point with the same name; the i-th group of road surface points with the same name is the road surface points with the same name corresponding to any group of road surface point cloud data with the same name. Based on the location of each group of road surface points with the same name in the target plane and the target elevation value in the target direction, a reference road surface point cloud map is constructed.

[0087] In practice, under dry road conditions, vehicles equipped with RTK-GPS and radar can repeatedly collect road surface point cloud data along roads (such as urban main roads and secondary roads), and the collection path can cover the entire road area.

[0088] The collected road surface point cloud data can then be processed as follows to obtain a reference road surface point cloud map: ① A statistical filtering algorithm is used to remove noise points from the road surface point cloud data collected multiple times. The number of neighboring points for denoising can be set to 20, and the standard deviation threshold can be set to 1.0. ② Use the ICP algorithm to perform same-name point matching on the road surface point cloud data collected multiple times to obtain the second matching result; optionally, the total number of collections can be 3; the number of iterations of the ICP algorithm can be set to 50, and the convergence threshold can be set to 0.001.

[0089] ③ Based on the second matching result, multiple road surface points located at the same position in the target plane are identified as a group of road surface points with the same name.

[0090] ④ For each group of road surface points with the same name, determine the target elevation value of the group of road surface points in the target direction based on the third elevation value of each road surface point in the target direction. Optionally, the average of the third elevation values ​​of each road surface point can be determined as the target elevation value of the group of road surface points with the same name; alternatively, the minimum of the third elevation values ​​of each road surface point can be determined as the target elevation value of the group of road surface points with the same name.

[0091] ⑤ Construct a reference road surface point cloud map. Store the positions of each group of road surface points with the same name in the target plane and the target elevation values ​​in the target direction to obtain the reference road surface point cloud map. Optionally, the data storage format in the reference road surface point cloud map can be point cloud data (PCD) format to support fast indexing and querying.

[0092] It is worth mentioning that, to improve the accuracy of constructing the reference road surface point cloud map, on the one hand, under good weather conditions, vehicle-mounted radar can be used to collect road surface point cloud data in a dry state, thus avoiding environmental interference with the accuracy of the collected point cloud data and ensuring the accuracy of the collected road surface point cloud data in a dry state. On the other hand, the vehicle can be controlled to repeatedly collect road surface point cloud data along the road at the target speed to avoid the collected road surface point cloud data being tilted due to vehicle posture. Alternatively, after obtaining multiple sets of road surface point cloud data with the same name, each set of road surface point cloud data can be corrected using the vehicle's motion posture information, and then the reference road surface point cloud map can be constructed using the corrected sets of road surface point cloud data. Optionally, the target driving speed can be obtained through calibration and can be 30 km / h.

[0093] This application embodiment employs an iterative nearest point algorithm to match corresponding points in multiple collections of dry road surface point cloud data, obtaining matching results and extracting multiple sets of corresponding road surface point cloud data; and for each set of corresponding points, the target elevation value of its target direction is determined by combining the third elevation value of each point, and then the reference road surface point cloud map is constructed based on the position of each set of corresponding points on the target plane and the target elevation value, which can improve the accuracy of the reference road surface point cloud map.

[0094] In one embodiment, step S104 above: determining the second water depth of the waterlogged area based on the first water depth and the vehicle's motion state information, includes: Based on the vehicle's motion status information, determine the correction amount for the water depth. The first water depth is corrected using the water depth correction factor to obtain the second water depth.

[0095] Optionally, the water depth correction amount can be obtained by querying the correspondence table between the motion state and the depth correction amount using the vehicle's motion state information; alternatively, the water depth correction amount can be calculated based on the vehicle's motion state information and a preset correction coefficient.

[0096] Optionally, the preset correction coefficients may include pitch angle correction coefficients, roll angle correction coefficients, and driving speed correction coefficients. Each correction coefficient can be obtained through actual vehicle calibration: within the full operating range of pitch angle, roll angle, and driving speed, several sets of data on actual water depth and measured water depth are collected, and each correction coefficient is obtained by fitting using a multiple linear regression algorithm, which is used to compensate for the dynamic effects of the water surface and measurement errors caused by changes in vehicle attitude and driving speed.

[0097] In practice, the water depth correction can be calculated using formula (1): (1) in, This is a correction amount for the water depth. , These are the vehicle's pitch angle, roll angle, and speed, respectively. , , These are the pitch angle correction factor, roll angle correction factor, and speed correction factor, which can be optionally... , , .

[0098] Optionally, the water depth correction can be expressed as an absolute value or a relative proportion. When expressed as an absolute value, the water depth correction can be added to the first water depth to obtain the second water depth; when expressed as a relative proportion, the water depth correction can be multiplied by the first water depth to obtain the second water depth.

[0099] This application embodiment determines the water depth correction amount based on the vehicle's motion state information; and uses the water depth correction amount to correct the first water depth to obtain the second water depth. This can effectively reduce the water depth estimation error caused by driving disturbances and improve the estimation accuracy of road water depth.

[0100] In one embodiment, after step S104 above: determining the second water depth of the waterlogged area based on the first water depth and the vehicle's motion state information, the road water depth estimation method provided in this application embodiment further includes: Based on the comparison results between the second water depth and the water depth threshold, the water accumulation risk level of the water accumulation area is determined; Based on the level of flood risk, control the vehicle to perform corresponding evasive maneuvers.

[0101] In practice, the current depth range of the second water accumulation depth can be determined based on the comparison between the second water accumulation depth and the water accumulation depth threshold. Then, using the current depth range, the correspondence between preset depth ranges and risk levels can be queried to obtain the water accumulation risk level of the water-filled area. Optionally, the preset correspondence between depth ranges and risk levels can be as follows: ① Low risk level: water depth This risk level indicates that the water accumulation in the flooded area is shallow, has little impact on the passability of most vehicles, and basically does not affect the normal operation of key vehicle components (such as engines and electrical systems). ② Medium risk level: This risk level indicates that the water depth in the flooded area is moderate, and there is a potential risk of vehicles wading through water, which may cause water to splash onto the vehicle chassis, get the electrical system wet, and some vehicles with low chassis may have difficulty passing through. ③ High-risk level: This risk level indicates that the water in the flooded area is deep, which can easily lead to serious malfunctions such as engine stalling due to water ingress, vehicle floating, and electrical system short circuits, endangering driving safety.

[0102] The above h Indicates the second depth of accumulated water; , The threshold for water depth. ;Optionally, , Optionally, the water depth threshold can also be determined based on vehicle information, meaning that different water depth thresholds can be set for different vehicles.

[0103] Furthermore, the determined flood risk level can be used to query the preset correspondence between the risk level and vehicle avoidance actions, and the vehicle can be controlled to perform the corresponding avoidance actions. Optionally, the avoidance actions may include at least one type of avoidance action such as outputting avoidance prompts, performing steering, or braking.

[0104] Optionally, the preset risk level and the correspondence between vehicle avoidance actions can be as follows: When the water accumulation risk level is medium or high, the vehicle will trigger an audible and visual warning to alert the driver to the risk of water accumulation. When the water accumulation risk level is high, the vehicle's motion status information can be combined to output decision suggestions such as deceleration and detour to ensure the vehicle's driving safety.

[0105] It is worth mentioning that, after step S104 above: determining the second water depth of the waterlogged area based on the first water depth and the vehicle's motion state information, the road water depth estimation method provided in this application embodiment may further include: Output at least one of the following road flooding depth estimation results: ① Water accumulation area segmentation mask; the mask can be a binarized image with the same resolution as the road surface image. The white area (pixel value of 1) in the mask can represent the water accumulation area, and the black area (pixel value of 0) in the mask can represent the non-water accumulation area. ② Water depth distribution map; In actual implementation, the second water depth of each water accumulation area can be mapped to a pseudo-color image, with different colors corresponding to different water depth ranges, thus obtaining a water depth distribution map; ③ The average and maximum water depth values ​​of each waterlogged area; ④ Flood risk level of each flooded area; In practice, the flood risk level of each flooded area can be indicated by text and / or color, such as green for low flood risk, yellow for medium flood risk, and red for high flood risk.

[0106] This application embodiment determines the water accumulation risk level of the water accumulation area based on the comparison result between the second water accumulation depth and the water accumulation depth threshold; and controls the vehicle to perform corresponding avoidance operations based on the water accumulation risk level, which can significantly reduce the vehicle's water wading risk and thus improve the vehicle's driving safety.

[0107] Optionally, the aforementioned vehicle-mounted camera can be an optical camera; the aforementioned vehicle-mounted radar can be a 4D millimeter-wave radar. Please refer to... Figure 2 and Figure 3In one specific embodiment, the road water depth estimation method provided in this application may further include, but is not limited to, the following steps: (1) Multimodal data synchronous acquisition and preprocessing: 1) Data acquisition: RGB images of the road scene are acquired in real time through the vehicle-mounted optical camera; 3D point cloud data of the road, radar reflection intensity and target speed information are acquired through the 4D millimeter-wave radar; the vehicle-mounted IMU sensor is activated to acquire vehicle pitch angle and roll angle, the RTK-GPS module is used to acquire the real-time position coordinates of the vehicle, and the wheel speed and driving speed are acquired through the wheel speed sensor. All sensor data are stored in the vehicle-mounted edge computing unit, with reserved timestamp and device identification fields. 2) Time synchronization: A unified trigger pulse signal is generated using the synchronization trigger module and connected to the trigger interfaces of the camera and the millimeter-wave radar respectively, controlling the two types of sensors to start data acquisition synchronously when the trigger signal is received. 3) Spatial alignment: Using the pre-calibrated intrinsic parameter matrix of the camera and the extrinsic parameter matrix between the camera and the radar, the reflection intensity and distance information of the road 3D point cloud are projected onto the image coordinate system to generate a two-channel point cloud projection feature map.

[0108] (2) Lightweight water body segmentation based on radar prior guidance: 1) Utilizing the low reflectivity of water bodies by millimeter-wave radar, the three-dimensional point cloud data of the road is traversed to select point cloud sets with reflectivity below the reflectivity threshold. The point cloud sets are then projected onto the image coordinate system to generate a binary water body candidate mask as the prior guidance for the model; 2) The aligned RGB image, the point cloud projection feature map, the water body candidate mask, and the labeled water body segmentation mask (which can be obtained through manual annotation) are input into the trained target neural network model. The target neural network model can extract visual features and radar reflection features based on the input data, and fuse and segment the extracted features to output the water body segmentation mask. ,in Represents pixels in an RGB image For water bodies, 0 represents a pixel. This is a non-water area.

[0109] (3) Water depth estimation based on point cloud registration and dynamic compensation: 1) Extracting three-dimensional point cloud data of water accumulation area: The three-dimensional point cloud data of the road collected by radar is corrected in position and attitude and matched with the reference road surface point cloud map to obtain the aligned road surface point cloud. 2) Prior map matching and elevation difference calculation: in water body segmentation mask Extraction within the corresponding water body area Each pixel Corresponding 3D points In the direction perpendicular to the RGB image z Coordinates along the axis As a three-dimensional point The first elevation value is obtained by referring to the point cloud map. Query the ideal road table point corresponding to this pixel. z Coordinates along the axis , obtain three-dimensional points The second elevation value is used to calculate the difference between the two to obtain the three-dimensional point. 3) Dynamic compensation for water depth: Using the water depth correction amount determined based on vehicle motion state information, the three-dimensional points are adjusted accordingly. The water depth was corrected to obtain three-dimensional points. The corrected water depth. In actual implementation, the three-dimensional points can be calculated using formula (2). Corrected water depth : (2) (4) Risk level classification and early warning information output: 1) Extract vehicle information: Determine the water depth threshold based on vehicle information; 2) Risk level determination: Determine the water risk level of the water area based on the comparison results of the corrected water depth of each three-dimensional point in the water area with the water depth threshold; 3) Early warning information output: Output risk avoidance prompts based on the water risk level of the water area.

[0110] The specific implementation process of this embodiment can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0111] Please see Figure 4 This application embodiment also provides a road water depth estimation device 400, which can implement the above-mentioned road water depth estimation method. The device 400 may include a water accumulation area determination module 401, a point cloud data determination module 402, a first water depth determination module 403, and a second water depth determination module 404.

[0112] The water accumulation area determination module 401 can be used to determine the water accumulation area of ​​the road based on the multimodal perception data of the road where the vehicle is located; the multimodal perception data includes the first road surface point cloud data; The point cloud data determination module 402 can be used to determine the actual point cloud data of the waterlogged area based on the first road surface point cloud data of the road. The first water depth determination module 403 can be used to determine the first water depth of the waterlogged area based on the actual road surface point cloud data and the ideal road surface point cloud data of the waterlogged area; the ideal road surface point cloud data is determined based on the road surface point cloud data of the waterlogged area in a dry state. The second water depth determination module 404 can be used to determine the second water depth of the water accumulation area based on the first water depth and the vehicle's motion state information.

[0113] The road water depth estimation device provided in this application embodiment can implement all the steps of the above-described road water depth estimation method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0114] This application also provides a road water depth estimation device, including a processor and a memory. The memory stores a program or instructions that can run on the processor. When the program or instructions are executed by the processor, they implement the various steps of the above-described road water depth estimation method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0115] Figure 5 To illustrate the hardware structure of the road waterlogging depth estimation device according to this application embodiment, the road waterlogging depth estimation device includes: The processor 501 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 502 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 using the road water depth estimation method of the embodiments of this application. The input / output interface 503 is used to implement information input and output; The communication interface 504 is used to enable communication and interaction between this road water depth estimation device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 505 transmits information between the various components of the road water depth estimation device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504); The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other via bus 505 within the road water depth estimation device.

[0116] The road water depth estimation device provided in this application embodiment can implement all the steps of the above-described road water depth estimation method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0117] This application also provides a computer-readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various steps of the above-described road water depth estimation method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0118] The processor is the processor in the road flooding depth estimation device of the above embodiment. The computer-readable storage medium includes computer-readable storage media such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0119] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various steps of the above-described road water depth estimation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0120] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0121] This application provides a computer program product stored in a storage medium. The program product is executed by at least one processor to implement the various steps of the road water accumulation depth estimation method embodiment described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0122] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not delete other identical elements present in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0124] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for estimating the depth of road flooding, characterized in that, include: Based on multimodal perception data of the road where the vehicle is located, the water accumulation area of ​​the road is determined; The multimodal sensing data includes the first path table point cloud data; Based on the first road surface point cloud data of the road, the actual road surface point cloud data of the waterlogged area is determined; Based on the actual road surface point cloud data and the ideal road surface point cloud data of the waterlogged area, the first waterlogging depth of the waterlogged area is determined; The ideal road surface point cloud data is determined based on the road surface point cloud data of the waterlogged area in a dry state; Based on the first water depth and the vehicle's motion status information, the second water depth of the water-filled area is determined.

2. The method as described in claim 1, characterized in that, The multimodal sensing data also includes road surface images; The determination of waterlogged areas on the road based on multimodal perception data of the road where the vehicle is located includes: A point cloud projection feature map is generated based on the projection result of the first road surface point cloud data of the road in the target image coordinate system; the target image coordinate system is the image coordinate system corresponding to the road surface image. The target data is input into the trained target neural network model to obtain a water body segmentation mask; the target data includes the road surface image and the point cloud projection feature map; the water body segmentation mask is used to characterize whether each pixel in the road surface image belongs to a water accumulation area. The water accumulation area is determined based on the water body segmentation mask.

3. The method as described in claim 2, characterized in that, The target data also includes a water body candidate mask; the water body candidate mask is used to guide the target neural network model to identify whether each pixel in the road surface image belongs to a waterlogged area; Before inputting the target data into the trained target neural network model to obtain the water segmentation mask, the method further includes: Based on the reflection intensity of each first path point in the first path point cloud, first path points related to water bodies are selected from the first path point cloud. The first path meter point cloud data includes the reflection intensity of each first path meter point; The water body candidate mask is generated based on the projection position of the first path point related to the water body in the target image coordinate system.

4. The method as described in claim 2, characterized in that, Before inputting the target data into the trained target neural network model to obtain the water segmentation mask, the method further includes: A training dataset is obtained; the training dataset includes multiple training samples; each training sample includes a sample road surface image, a sample point cloud projection feature map, and a labeled water body segmentation mask; at least some of the training samples are obtained under different environmental conditions. Input the training samples from the training dataset into the neural network model to be trained to obtain the predicted water body segmentation mask; Based on the predicted water segmentation mask and the labeled water segmentation mask, determine the target loss function value of the neural network model to be trained; Based on the target loss function value, the network parameters of the neural network model to be trained are updated to train the target neural network model.

5. The method as described in claim 2, characterized in that, The target neural network model includes an encoder and a decoder; The step of inputting the target data into the trained target neural network model to obtain the water segmentation mask includes: The encoder performs downsampling and feature extraction on the target data to obtain the deep semantic features of the target data; The deep semantic features are upsampled by the decoder, and the upsampled features are fused with the features extracted by the encoder to obtain the water body segmentation mask.

6. The method as described in claim 1, characterized in that, The first road table point cloud data was obtained through vehicle-mounted radar; The determination of the actual road surface point cloud data of the waterlogged area based on the first road surface point cloud data of the road includes: Based on the vehicle's motion attitude information and the transformation relationship between the vehicle's inertial measurement unit coordinate system and the vehicle's radar coordinate system, the first road surface point cloud data is corrected to obtain the second road surface point cloud data of the road; the motion attitude information is acquired by the vehicle's inertial measurement unit. Based on the second road surface point cloud data of the road, the actual road surface point cloud data of the waterlogged area is determined.

7. The method as described in claim 1, characterized in that, The first road surface point cloud data is acquired by vehicle-mounted radar; the ideal road surface point cloud data is point cloud data in a global geographic coordinate system. The determination of the actual road surface point cloud data of the waterlogged area based on the first road surface point cloud data of the road includes: Based on the vehicle's location information in the global geographic coordinate system and the conversion relationship between the vehicle radar coordinate system and the global geographic coordinate system, the first road surface point cloud data is converted to obtain the third road surface point cloud data of the road. Based on the third road surface point cloud data of the road, the actual road surface point cloud data of the waterlogged area is determined.

8. The method as described in claim 7, characterized in that, The determination of the actual road surface point cloud data of the waterlogged area based on the third road surface point cloud data of the road includes: Using the iterative nearest point algorithm, the third road surface point cloud data and the reference road surface point cloud map are matched for corresponding points to obtain a first matching result; the corresponding point matching is used to match road surface points located at the same position in the target plane under the global geographic coordinate system; the target plane is parallel to the road surface; the reference road surface point cloud map is constructed based on the ideal road surface point cloud data; Based on the first matching result, the position of each third table point in the third table point cloud data in the target plane is updated to obtain the fourth table point cloud data of the road; By filtering road surface point cloud data related to the waterlogged area from the fourth road surface point cloud data of the road, the actual road surface point cloud data of the waterlogged area is obtained.

9. The method as described in claim 1, characterized in that, The actual road surface point cloud data includes the positions of multiple actual road surface points in the target plane and the first elevation value in the target direction; The ideal road surface point cloud data includes the second elevation values ​​of multiple ideal road surface points in the target direction; The target direction is perpendicular to the road surface; The determination of the first water depth in the waterlogged area based on the actual road surface point cloud data and the ideal road surface point cloud data includes: For each actual road surface point, based on the position of the actual road surface point in the target plane, a reference road surface point cloud map is queried to obtain the second elevation value of the target road surface point; the target road surface point is an ideal road surface point whose position in the target plane matches the position of the actual road surface point. The water depth at the actual road surface point is determined based on the difference between the first elevation value and the second elevation value. Based on the water depth at each of the actual road surface points, the first water depth of the waterlogged area is determined.

10. The method as described in claim 9, characterized in that, Before obtaining the second elevation value of the target road surface point by querying a reference road surface point cloud map based on the position of the actual road surface point in the target plane, the method further includes: Using the iterative nearest point algorithm, the road surface point cloud data of the road under dry conditions collected multiple times are matched with corresponding points to obtain a second matching result; Based on the second matching result, multiple sets of road surface point cloud data with the same name are obtained; each set of road surface point cloud data with the same name includes a third elevation value of a set of road surface points in the target direction; the road surface points with the same name are used to characterize road surface points that are in the same position in the target plane; For the i-th group of road point corresponding to the same name, the target elevation value of the i-th group of road point corresponding to the same name is determined in the target direction based on the third elevation value of each road point corresponding to the same name; the i-th group of road point corresponding to the same name is any group of road point cloud data corresponding to the same name. Based on the position of each group of road surface points with the same name in the target plane and the target elevation value in the target direction, the reference road surface point cloud map is constructed.

11. The method as described in claim 1, characterized in that, Determining the second water depth of the waterlogged area based on the first water depth and the vehicle's motion state information includes: Based on the vehicle's motion state information, the water depth correction amount is determined; The first water depth is corrected using the water depth correction amount to obtain the second water depth.

12. The method as described in claim 1, characterized in that, After determining the second water depth of the waterlogged area based on the first water depth and the vehicle's motion state information, the method further includes: Based on the comparison between the second water depth and the water depth threshold, the water accumulation risk level of the water accumulation area is determined; Based on the level of water accumulation risk, the vehicle is controlled to perform corresponding evasive maneuvers.

13. A device for estimating the depth of road flooding, characterized in that, The device includes a water accumulation area determination module, a point cloud data determination module, a first water accumulation depth determination module, and a second water accumulation depth determination module. The water accumulation area determination module is used to determine the water accumulation area of ​​the road based on multimodal perception data of the road where the vehicle is located; the multimodal perception data includes first road surface point cloud data; The point cloud data determination module is used to determine the actual point cloud data of the waterlogged area based on the first road surface point cloud data of the road. The first water depth determination module is used to determine the first water depth of the water accumulation area based on the actual road surface point cloud data and the ideal road surface point cloud data of the water accumulation area; The ideal road surface point cloud data is determined based on the road surface point cloud data of the waterlogged area in a dry state; The second water depth determination module is used to determine the second water depth of the water accumulation area based on the first water depth and the vehicle's motion state information.

14. A road flooding depth estimation device, characterized in that, The road water depth estimation device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 12.

Images