Method and device for determining prompt information in vehicle, vehicle and storage medium

Abstract

The application discloses a method and device for determining prompt information in a vehicle, a vehicle and a storage medium. The method comprises: in response to the vehicle moving to a water area, acquiring multi-modal data of an environment where the water area is located; determining a current state vector corresponding to the water area based on the multi-modal data, wherein the current state vector is used to represent the physical characteristics of the water area at the current time; based on the current state vector, at least one future state vector of the water area at at least one future time is predicted, wherein the future state vector is used to represent the physical characteristics of the water area at the future time, and the future time is later than the current time; and determining prompt information based on the future state vector, wherein the prompt information is used to prompt the risk of the vehicle driving in the water area. The application solves the technical problem of low risk assessment accuracy in the process of the vehicle passing.

Description

Technical Field

[0001] This application relates to the field of vehicles, and more specifically, to a method, apparatus, vehicle, and storage medium for determining prompt information in a vehicle. Background Technology

[0002] Currently, the main methods for vehicles to pass through in adverse weather and complex road conditions are to use radar ranging and visual matching maps to perceive the traffic environment, and then compare and compensate the results obtained from radar ranging and visual matching maps to ultimately determine whether the vehicle can pass.

[0003] However, the above methods focus on a single perception dimension. Furthermore, the entire perception process is more focused on the current moment and ignores changes in the situation in future moments. Therefore, the technical problem of low accuracy in risk assessment during vehicle passage remains.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method, apparatus, vehicle, and storage medium for determining warning information in a vehicle, in order to at least solve the technical problem of low accuracy in risk assessment during vehicle passage.

[0006] According to one aspect of the embodiments of this application, a method for determining warning information in a vehicle is provided. The method may include: in response to the vehicle moving to a wading area, acquiring multimodal data of the environment of the wading area, wherein the multimodal data includes modal data collected by multiple sensors in the vehicle at the current moment, and the multimodal data is used to characterize the environmental characteristics of the wading area; determining a current state vector corresponding to the wading area based on the multimodal data, wherein the current state vector is used to characterize the physical characteristics of the wading area at the current moment; predicting at least one future state vector of the wading area at at least one future moment based on the current state vector, wherein the future state vector is used to characterize the physical characteristics of the wading area at a future moment, and the future moment is later than the current moment; and determining warning information based on the future state vector, wherein the warning information is used to warn of the risks of driving in the wading area.

[0007] Furthermore, the method also includes: obtaining a historical state vector sequence corresponding to the water-traversed area, wherein the historical state vector sequence includes at least one historical state vector corresponding to a historical time period, and is used to characterize the changing trend of the physical characteristics of the water-traversed area during the historical time period, and the historical time period is earlier than the current time; predicting at least one future state vector of the water-traversed area at at least one future time based on the current state vector, including: determining a current status code based on the current state vector, and determining a trend code based on the historical state vector sequence, wherein the current status code is used to characterize the instantaneous state of the water-traversed area at the current time, and the trend code is used to characterize the changing trend of the historical state of the water-traversed area; and predicting a future state vector based on the current status code and the trend code.

[0008] Furthermore, based on multimodal data, the current state vector corresponding to the wading area is determined, including: identifying the multimodal data to obtain the vehicle's driving condition information; determining weight data that matches the driving condition information, wherein the weight data is used to characterize the degree of influence of different modal data in the multimodal data on physical properties; and using the weight data to transform the multimodal data to obtain the current state vector.

[0009] Furthermore, using weighted data, the multimodal data is transformed to obtain the current state vector, including: extracting features from the multimodal data to obtain multiple initial feature data; processing the initial feature data using a physical forward model to obtain an initial state vector, where the physical forward model is used to characterize the correlation between the initial feature data and the standard state vector; determining the differences between the multiple initial state vectors and the multiple standard state vectors to obtain multiple differences; transforming the multiple differences using weighted data to obtain an objective function, where the objective function is used to characterize the degree of error between the initial state vector and the standard state vector; and performing nonlinear adjustment on the objective function to obtain the current state vector.

[0010] Furthermore, based on the current state vector, the current state code is obtained, including: encoding the current state vector and the initial state vector to obtain the current state code.

[0011] Furthermore, based on the current status code and trend code, the future state vector is predicted, including: fusing the current status code and trend code to obtain fused features; identifying the fused features to determine the future state vector; the method also includes: identifying the fused features to determine the traversability index of the water crossing area, wherein the traversability index is used to characterize the probability of a vehicle safely passing through the water crossing area.

[0012] According to one aspect of the embodiments of this application, a device for determining warning information in a vehicle is provided. The device may include: an acquisition unit, configured to acquire multimodal data of the environment of the wading area in response to the vehicle moving to a wading area, wherein the multimodal data includes modal data collected by multiple sensors in the vehicle at the current moment, and the multimodal data is used to characterize the environmental characteristics of the wading area; a processing unit, configured to determine a current state vector corresponding to the wading area based on the multimodal data, wherein the current state vector is used to characterize the physical characteristics of the wading area at the current moment; a prediction unit, configured to predict at least one future state vector of the wading area at at least one future moment based on the current state vector, wherein the future state vector is used to characterize the physical characteristics of the wading area at a future moment, and the future moment is later than the current moment; and a determination unit, configured to determine warning information based on the future state vector, wherein the warning information is used to warn of the risks of driving in the wading area.

[0013] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program runs the methods of various embodiments of this application.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods in various embodiments of this application.

[0018] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0019] In this embodiment, in response to a vehicle moving to a wading area, multimodal data of the environment in the wading area is acquired. This multimodal data includes modal data collected by multiple sensors in the vehicle at the current moment, and is used to characterize the environmental characteristics of the wading area. Based on the multimodal data, a current state vector corresponding to the wading area is determined, where the current state vector characterizes the physical characteristics of the wading area at the current moment. Based on the current state vector, at least one future state vector of the wading area at at least one future moment is predicted, where the future state vector characterizes the physical characteristics of the wading area at a future moment, which is later than the current moment. Based on the future state vector, a warning message is determined, where the warning message warns of the risks of driving in the wading area. In other words, in this embodiment, the physical characteristics of the wading area are determined based on modal data collected by multiple sensors. Based on the aforementioned multimodal data (i.e., multiple modal data), the physical characteristics of the corresponding wading area at future times are predicted. Corresponding prompt information is generated based on the predicted future state vector. Since the above method predicts the changing trend through multimodal data obtained by multiple sensors, it overcomes the situation where relying solely on a single water depth information or other single perception dimension leads to the inability to perceive the multidimensional physical state of the wading area and the lack of future trend prediction ability. It achieves the goal of continuous, multidimensional, and forward-looking assessment of the wading area, thereby improving the technical effect of risk assessment accuracy during vehicle passage and solving the technical problem of low risk assessment accuracy during vehicle passage. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a schematic diagram of a method for determining prompt information in a vehicle according to an embodiment of this application;

[0022] Figure 2 This is a flowchart illustrating the overall operation of an optional sensing system according to an embodiment of this application;

[0023] Figure 3 This is a flowchart of an optional water body multi-attribute joint inversion engine algorithm based on a physical model, according to an embodiment of this application;

[0024] Figure 4 This is a flowchart of an optional "current-trend" dual-stream temporal prediction fusion network architecture according to an embodiment of this application;

[0025] Figure 5This is a schematic diagram of an optional system decision output and human-computer interaction interface according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of a device for determining prompt information in a vehicle according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] According to an embodiment of this application, a method embodiment for determining prompt information in a vehicle is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] Figure 1 This is a schematic diagram of a method for determining prompt information in a vehicle according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps.

[0031] Step S102: In response to the vehicle moving to the wading area, acquire multimodal data of the environment in the wading area.

[0032] In the technical solution provided by step S102 of this application embodiment, the aforementioned wading area can be a road surface area covered by water along the vehicle's driving path. The aforementioned multimodal data can be used to characterize the environmental characteristics of the wading area, such as water depth, water optical transparency, dynamic fluctuations of the water surface, and the physical dielectric properties of the underwater substrate. The multimodal data can include multiple modal data collected by multiple sensors in the vehicle at the current moment. The aforementioned multimodal data may include, but is not limited to, images acquired by a camera, point cloud intensity distribution and pulse broadening data returned by a lidar, polarization differential reflectivity and co-polarization correlation coefficient of a millimeter-wave radar, etc. The aforementioned environmental characteristics can refer to physically significant multidimensional water and surface state parameters that affect the safety of vehicle wading. The aforementioned sensors can be multiple different types of sensors, such as vehicle-mounted cameras and vehicle-mounted lidar.

[0033] Optionally, when the vehicle moves to the wading area, multimodal data of the environment in the wading area can be obtained through multiple sensors deployed in the vehicle.

[0034] Optionally, in response to the vehicle moving to the wading area, optical images of the wading area can be collected by the vehicle's onboard camera to obtain water surface reflection characteristics and visual texture information; point cloud data of the wading area can be collected by the vehicle's onboard lidar; electromagnetic echo signals of the wading area can be collected by the vehicle's onboard millimeter-wave radar, etc.; the different data collected by the above-mentioned different sensors are spatiotemporally aligned to form multimodal data that synchronously characterizes the environmental characteristics of the wading area at the current moment.

[0035] In this embodiment of the application, step S102 above can realize the synchronous collection and structured expression of multi-source environmental information of the water-related area at the current moment, providing basic data support for subsequent steps.

[0036] Step S104: Based on multimodal data, determine the current state vector corresponding to the water-contaminated area.

[0037] In the technical solution provided by step S104 in the embodiments of this application, the current state vector can be used to represent the physical characteristics of the water-contaminated area at the current moment. It can be the optimal state estimate, physical state estimate, or optimal physical state estimate. The aforementioned physical characteristics can refer to physical quantities such as water depth, turbidity, water surface ripple intensity, and underwater substrate dielectric constant of the wading area.

[0038] Optionally, after obtaining multimodal data of the environment in which the water-involved area is located, the current state vector corresponding to the water-involved area can be determined based on the multimodal data.

[0039] Optionally, the turbidity and water surface wave intensity can be calculated by combining the optical images collected by the vehicle-mounted camera with the water surface reflection characteristics and visual texture information in the multimodal data; the water depth and turbidity can be estimated by the point cloud data collected by the vehicle-mounted lidar; the underwater substrate dielectric constant can be determined by the electromagnetic echo signal collected by the vehicle-mounted millimeter-wave radar; and the current state vector containing water depth, turbidity, water surface wave intensity and underwater substrate dielectric constant can be output by optimizing the solution using a pre-set physical model.

[0040] In this embodiment of the application, step S104 above can realize the synchronous, interpretable, and non-black-box quantitative expression of the multidimensional physical state of the water-related area, breaking through the limitation of only outputting a single water depth parameter, and providing a physically meaningful input basis for subsequent processing.

[0041] Step S106: Based on the current state vector, predict at least one future state vector of the water-crossing area at at least one future time.

[0042] In the technical solution provided by step S106 in the embodiments of this application, the future state vector can be used to characterize the physical characteristics of the water-contaminated area at a future time, where the future time is later than the current time.

[0043] Optionally, after determining the current state vector corresponding to the water-bound area, at least one future state vector of the water-bound area at at least one future time can be predicted based on the current state vector.

[0044] Optionally, based on the current state vector, the sequence of current state vectors from past moments can be encoded using a temporal convolutional network to extract features of the evolution of physical properties of the water-bound area over time (e.g., a continuous upward trend in water depth, periodic increase in wave amplitude, and a slow decrease in substrate dielectric constant). The current state vector can be encoded using a multilayer perceptron. The features obtained by encoding the current state vector based on the aforementioned features of the evolution of physical properties over time can be used to predict the future state vectors of water depth, turbidity, surface wave intensity, and underwater substrate dielectric constant at future moments (e.g., 2 to 3 seconds).

[0045] In this embodiment of the application, step S106 described above can enable forward-looking dynamic prediction of the physical state of water-related areas, providing a key early warning window and significantly improving the initiative and safety of decision-making.

[0046] Step S108: Determine the prompt information based on the future state vector.

[0047] In the technical solution provided by step S108 of this application embodiment, the aforementioned prompting information can be used to characterize the comprehensive traffic risk level and evolution trend of the water-crossing area at future times. This prompting information is used to alert vehicles to the risks of driving in water-crossing areas and can be text or image information. It can be displayed on the vehicle's human-machine interface, such as in an augmented reality head-up display (AR-HUD) / dashboard, and may include, but is not limited to, information such as visualization of multi-dimensional perception results, risk prediction prompts, and graded decision-making suggestions. It should be noted that this is merely an example and does not impose specific limitations on the type of prompting information.

[0048] Optionally, after predicting at least one future state vector of the water-crossing area at at least one future time based on the current state vector, the prompt information can be determined based on the future state vector.

[0049] Optionally, based on the predicted values ​​of various physical quantities (such as water depth, turbidity, wave intensity, and substrate dielectric constant) contained in the future state vector, a pre-defined multidimensional risk threshold model is used to quantitatively assess the risk level and determine the corresponding warning information. The model type of the aforementioned multidimensional risk threshold model may include, but is not limited to, a continuous risk index model weighted by physical equations, or a fuzzy inference model based on physical constraints.

[0050] Optionally, the aforementioned prompts may include, but are not limited to, visualization of multi-dimensional perception results, risk prediction prompts, and tiered decision-making suggestions. These can be output in multiple modalities, such as visualizing water depth heat maps, dynamic surge prediction animations, and substrate hardness distribution on an augmented reality head-up display (AR-HUD); for example, providing voice prompts such as "The water ahead is too deep and the bottom is slippery. Oncoming vehicles will cause large waves. Please detour immediately"; for example, sending a "enter wading mode" command to the power domain via control commands (raising the air intake, limiting torque, and switching drive modes); or for example, requesting a high-precision map system to replan the detour route via navigation linkage.

[0051] Through the steps S102 to S108 described above, in response to the vehicle moving to the wading area, multimodal data of the environment in the wading area can be acquired. This multimodal data includes modal data collected by multiple sensors in the vehicle at the current moment, and is used to characterize the environmental characteristics of the wading area. Based on the multimodal data, a current state vector corresponding to the wading area is determined, where the current state vector characterizes the physical characteristics of the wading area at the current moment. Based on the current state vector, at least one future state vector of the wading area at at least one future moment is predicted, where the future state vector characterizes the physical characteristics of the wading area at a future moment, which is later than the current moment. Based on the future state vector, a warning message is determined, where the warning message indicates the risks of driving in the wading area. In other words, in this embodiment, the physical characteristics of the wading area are determined based on modal data collected by multiple sensors. Based on the aforementioned multimodal data (i.e., multiple modal data), the physical characteristics of the corresponding wading area at future times are predicted. Corresponding prompt information is generated based on the predicted future state vector. Since the above method predicts the changing trend through multimodal data obtained by multiple sensors, it overcomes the situation where relying solely on a single water depth information or other single perception dimension leads to the inability to perceive the multidimensional physical state of the wading area and the lack of future trend prediction ability. It achieves the goal of continuous, multidimensional, and forward-looking assessment of the wading area, thereby improving the technical effect of risk assessment accuracy during vehicle passage and solving the technical problem of low risk assessment accuracy during vehicle passage.

[0052] The embodiments of this application will be described in detail below with reference to the steps described above.

[0053] As an optional implementation, the method further includes: obtaining a historical state vector sequence corresponding to the water-traversed area, wherein the historical state vector sequence includes at least one historical state vector corresponding to a historical time period, and is used to characterize the changing trend of the physical characteristics of the water-traversed area during the historical time period, and the historical time period is earlier than the current time; predicting at least one future state vector of the water-traversed area at at least one future time based on the current state vector, including: determining a current status code based on the current state vector, and determining a trend code based on the historical state vector sequence, wherein the current status code is used to characterize the instantaneous state of the water-traversed area at the current time, and the trend code is used to characterize the changing trend of the historical state of the water-traversed area; and predicting a future state vector based on the current status code and the trend code.

[0054] In this embodiment, the aforementioned historical state vector sequence can be used to characterize the changing trend of the physical characteristics of the water-traversed area over a historical period, and can also be referred to as a historical state sequence (which can be represented by S_t). The aforementioned current status encoding can be used to characterize the semantic compression and feature abstraction of the current state vector, and the extracted feature combination with a certain discriminative power for risk assessment can be represented by z_curr. The aforementioned trend encoding can be used to characterize the changing trend of the historical state of the water-traversed area, and can be represented by z_temp.

[0055] Optionally, a multi-dimensional physical state vector corresponding to the water area at each sampling time is continuously collected and stored at a fixed sampling frequency (e.g., 10 Hz) within a predetermined time window (e.g., the past N seconds, N∈[3,10]). The aforementioned historical state vector includes physical quantities such as water depth, optical turbidity, water surface wave intensity, and underwater substrate dielectric constant.

[0056] Optionally, the current state vector at the current moment is input into a lightweight multilayer perceptron encoder, which is compressed into a low-dimensional semantic vector, i.e., the current state code, through nonlinear mapping; the historical state vector sequence is input into a temporal convolutional network, which captures long-term dependencies through a multilayer causal convolutional structure, extracts the local and global trend patterns of state evolution over time, and outputs the trend code.

[0057] Optionally, the current status code is used as the query and the trend code is used as the key and value to calculate the attention weight matrix; the trend code is weighted and summed using the attention weight to obtain the fusion feature; the fusion feature is input into a preset fully connected prediction network to output a multi-dimensional state vector prediction sequence for future time, i.e., the future state vector.

[0058] For example, a Temporal Convolutional Network (TCN) can be used to process historical state sequences (represented by S_t) to capture dynamic patterns, and the output trend code can be represented by z_temp. This trend code can be used to characterize how four physical quantities—water depth, turbidity, undulation, and substrate hardness—have changed over the past N seconds. This trend code is not a future value, but rather a summary of historical dynamics.

[0059] In the embodiments of this application, the above method can be used to construct a forward-looking water risk prediction framework with physical interpretability and spatiotemporal dynamic perception capabilities, thereby realizing a decision-making shift from passive perception to proactive foresight.

[0060] As an optional implementation method, the current state vector corresponding to the wading area is determined based on multimodal data, including: identifying the multimodal data to obtain the vehicle's driving condition information; determining weight data that matches the driving condition information, wherein the weight data is used to characterize the degree of influence of different modal data in the multimodal data on physical characteristics; and using the weight data to transform the multimodal data to obtain the current state vector.

[0061] In this embodiment, the aforementioned driving condition information can characterize the dynamic operating state of the vehicle in a wading environment and its external interaction with the surrounding environment, and can be used to determine the vehicle's driving conditions. These driving conditions may include, but are not limited to, daytime, clear water, and tunnels. The aforementioned weighting data can characterize the degree of influence of different modal data in the multimodal data on physical characteristics.

[0062] Optionally, parameters such as vehicle speed, longitudinal acceleration, yaw rate, steering wheel angle, engine speed, and gear status are acquired via the vehicle bus; ambient light intensity and the presence of oncoming headlights are analyzed using images acquired by the vehicle's cameras; the vehicle's pitch and roll angles are acquired via the vehicle acceleration sensor to determine the impact of non-uniform buoyancy of the water on the vehicle body; and the above series of parameters are input into a lightweight operating condition classifier (e.g., a decision tree or a small MLP) to output discrete or continuous driving condition labels, i.e., the vehicle's driving condition information.

[0063] Optionally, a mapping table between driving conditions and weight data is pre-set, where each type of driving condition information corresponds to a set of weight data; the weight data is obtained by looking up the table based on the identified driving condition.

[0064] For example, the above weight data can be determined by the following complete decision representation example table. Table 1 is the complete decision representation example table. As shown in Table 1, if the driving condition is determined to be daytime, clear water, hard low, then the dominant sensor can be determined to be radar. Then the matching weight data under this driving condition can be determined to be: W=[w_vis,w_lidar,w_radar,w_prior], which is [0.6,0.2,0.15,0.05].

[0065] Table 1. Examples of Complete Decision Representations

[0066]

[0067] Optionally, the weight data can be represented by W. The content that the weight data may include can be set and selected according to actual needs. For example, the weight data can also be W=[w_vis,w_lidar,w_radar], where w_vis can be used to represent the confidence weight corresponding to the visual modality, w_lidar can be used to represent the confidence weight corresponding to the lidar modality, and w_radar can be used to represent the confidence weight corresponding to the millimeter-wave radar modality.

[0068] For example, data from each sensor is spatiotemporally aligned and then fed into the feature extraction data path in parallel. The algorithm fuses observations of the same scene from multiple sensors, such as a camera identifying bright areas due to specular reflection; a lidar point cloud in the corresponding area being sparse but containing backscattering points; and a millimeter-wave radar point cloud partially located on the water surface and partially penetrating the water. The algorithm ultimately fuses these observations to generate a high-confidence wading area. Specifically, the visual data processing path can output a water surface segmentation mask, texture features, and a visual turbidity index based on HSV space (represented by T_vis); the lidar data processing path can output echo intensity statistics and average pulse width (represented by W_mean) of the water surface area point cloud; the millimeter-wave radar data processing path can output polarization differential reflectivity (represented by Zdr), co-polarization correlation coefficient (represented by ρ_hv), and Doppler spectral width; and the dynamic compensation module receives all features and outputs real-time changing confidence weights.

[0069] Optionally, if the camera identifies the surface as water but the LiDAR point cloud density is not low (not meeting specular reflection requirements), the visual weight w_vis is automatically reduced while the LiDAR weight is increased. This mechanism directly ensures the system's robustness in the event of sensor failure.

[0070] Optionally, feature vectors are extracted from each modal data; the feature vectors are normalized to the same dimension according to the modality to form the original feature matrix; the features of each modality are weighted and fused to generate a fused feature vector; the fused feature vector is input into a physical constraint mapping network (e.g., a lookup table interpolation model) to map the fused features into the final current state vector.

[0071] In this embodiment, the above method enables the transformation of sensor fusion from static weighting to context-aware dynamic fusion, thereby improving the system's robustness in extreme environments.

[0072] As an optional implementation, multimodal data is transformed using weighted data to obtain the current state vector. This includes: extracting features from the multimodal data to obtain multiple initial feature data; processing the initial feature data using a physical forward model to obtain an initial state vector, wherein the physical forward model is used to characterize the correlation between the initial feature data and the standard state vector; determining the differences between the multiple initial state vectors and the multiple standard state vectors to obtain multiple differences; transforming the multiple differences using weighted data to obtain an objective function, wherein the objective function is used to characterize the degree of error between the initial state vector and the standard state vector; and performing nonlinear adjustment on the objective function to obtain the current state vector.

[0073] In this embodiment, the aforementioned initial feature data can be used to characterize the raw response signals observed by each sensor mode to the physical characteristics of the water-bound area. The aforementioned initial state vector can be used to characterize the initial estimation results based on a single sensor or unweighted fusion. The aforementioned physical forward model can be used to characterize the correlation between the initial feature data and the standard state vector. The aforementioned standard state vector can be used to characterize the theoretically optimal set of state estimates derived from the physical model under ideal noise-free and interference-free conditions. The aforementioned objective function can be a nonlinear weighted least squares cost function.

[0074] Optionally, the visual turbidity index is extracted from the visual modal data in the multimodal data; the average echo intensity of the point cloud in the water surface area is calculated for the lidar modal data; the co-polarization correlation coefficient is extracted for the millimeter-wave radar modal data; and for each sensor modality, the corresponding physical forward model is called to back-map the observed features into a set of preliminary estimated physical states.

[0075] Optionally, a set of standard state vectors is set; for the initial state vector of each sensor, the weighted Euclidean difference between the initial state vector and the standard state vector is calculated.

[0076] Optionally, a modal weight vector is set; the residual vector of each sensor is weighted according to the corresponding weight to construct a global weighted error function, which is also a nonlinear weighted least squares cost function.

[0077] Optionally, based on a nonlinear optimization algorithm, the Jacobian matrix of the above function is calculated, and the optimal solution is output as the current state vector when the preset conditions are met or the number of iterations exceeds the limit.

[0078] For example, a joint inversion engine based on a physical model defines a physical state vector to be solved, which can be expressed by the following formula:

[0079] θ=[H,τ,σ_s,ε_r]

[0080] Where H can represent water depth, τ can represent turbidity, σ_s can represent wave roughness, and ε_r can represent substrate dielectric constant.

[0081] Optionally, a forward model can be constructed: a physical equation is established between the initial feature data and the standard state vector (represented by θ) for each sensor. The lidar model predicts echo intensity and pulse broadening. The millimeter-wave radar model, based on electromagnetic wave scattering theory, predicts polarization observations. The vision model predicts image texture and brightness, correlated with τ and σ_s. The inversion problem can be formalized as a nonlinear weighted least squares problem and solved using the Levenberg-Marquardt (L_M) algorithm, outputting the optimal state estimate in one step. This method directly produces the beneficial effect of multidimensional, interpretable perception, with each state variable having a corresponding physical definition.

[0082] For example, the input can be a weighted multimodal feature vector and dynamic weights (w_vis, w_lidar, and w_radar); the input is a weighted observation feature vector from the multimodal feature extraction and dynamic fusion module; the physical state vector to be solved is defined as θ; the target inversion variable is defined as a four-dimensional physical state vector θ=[H,τ,σ_s,ε_r]; a forward physical observation model is constructed; a deterministic physical relationship model between observations and physical states is constructed for each type of sensor. All of these models are differentiable, continuous, and analytical functions, supporting gradient optimization; a weighted nonlinear least squares cost function is constructed. The optimal physical state estimate can be solved using optimization algorithms such as Levenberg-Marquardt with θ. express.

[0083] Optionally, feature extraction is performed on the multimodal data to obtain multiple initial feature data. The initial feature data are then processed using a physical forward model to obtain initial state vectors, such as the vector output by the camera's visual features (i.e., the real-time observation value F_vision), the feature vector output by the lidar (i.e., the real-time observation value F_lidar), and the feature vector output by the millimeter-wave radar (i.e., the real-time observation F_radar). The differences between these initial state vectors and their corresponding standard state vectors are determined to obtain multiple differences, i.e., observation residuals |F_obs-M(θ)|. The standard state vector can be the standard physical characteristic of the wading area under a given physical state, i.e., the standard value that the sensor should observe under the initial feature data. It can be the physical state vector to be solved, which can be the vector M_vis(θ) corresponding to the camera's visual feature vector, the vector M_lidar(θ) corresponding to the lidar feature vector, and the vector M_radar(θ) corresponding to the millimeter-wave radar feature vector. Using weighted data, multiple differences are transformed to obtain the objective function J(θ), where the objective function characterizes the degree of error between the initial state vector and the standard state vector. The objective function J(θ) can be calculated using the following formula:

[0084] J(θ)=w_vis ||F_vision-M_vis(θ)|| 2 +w_lidar ||F_lidar-M_lidar(θ)|| 2 +w_radar ||F_radar-M_radar(θ)|| 2 +λ R(θ)

[0085] Here, R(θ) can be used to characterize the regularization term. λ can be used to characterize the regularization coefficient.

[0086] Optimization algorithms such as Levenberg-Marquardt can be used to nonlinearly adjust the objective function to obtain the optimal physical state estimate θ. The above optimal physical state estimate θ It can be calculated using the following formula:

[0087] θ =argminJ(θ)

[0088] In the embodiments of this application, the above method can be used to achieve high-precision, adaptive, and interpretable inversion of multimodal sensing data into a full-dimensional environmental state with clear physical meaning.

[0089] As an optional implementation, the current state code is obtained based on the current state vector, including: encoding the current state vector and the initial state vector to obtain the current state code.

[0090] Optionally, the current state vector (which can be the four-dimensional physical state output by the physical joint inversion engine) is concatenated with the multimodal initial feature vector to form a fused feature vector; the concatenated fused feature is input into a lightweight multilayer perceptron encoder, which can be composed of 2 to 3 fully connected neural networks, with each layer followed by an activation function and layer normalization, and finally outputs the current state code.

[0091] For example, a multilayer perceptron can be used to encode the current multimodal raw features (i.e., the initial state vector) and the latest θ_t (i.e., the current state vector) to output the current state encoding z_curr.

[0092] In the embodiments of this application, the above method can be used to construct a characterization mechanism that integrates physically interpretable states and raw sensor observations, which can significantly improve the perception robustness, decision credibility and prediction accuracy of the water accessibility prediction system.

[0093] As an optional implementation method, the future state vector is predicted based on the current status code and the trend code, including: fusing the current status code and the trend code to obtain the fused features; identifying the fused features to determine the future state vector; the method also includes: identifying the fused features to determine the passability index of the water crossing area, wherein the passability index is used to characterize the probability of a vehicle safely passing through the water crossing area.

[0094] In this embodiment, the aforementioned fusion features can be used to characterize the causal relationship and synergistic effect between the current physical state of the water-related environment and its historical dynamic evolution trend. The aforementioned trafficability index can be used to characterize the likelihood of a vehicle safely passing through a water-related area, and can be a comprehensive trafficability index (PI).

[0095] Optionally, the current status code is used as the query vector, and the trend code is used as the key vector and value vector; attention weights are calculated through a cross-stream attention mechanism; and the trend codes are weighted and aggregated to generate fused features.

[0096] Optionally, the fused features are input into a multi-step prediction head, which can consist of 2 to 3 fully connected layers, each followed by ReLU activation and layer normalization, and the output layer is a linear mapping; every four consecutive output dimensions correspond to a physical state vector at a future time.

[0097] Optionally, the fused features are input into an independent but parameter-shared mobility prediction head. This independent but parameter-shared mobility prediction head shares the first two network parameters with the multi-step prediction head, with only the final output layer being a single neuron. A normalized mobility index is output, with a value range of [0,1][0,1], representing the overall probability that a vehicle can safely pass through the wading area in the current and next 2 to 3 seconds.

[0098] For example, we can use the current state code as the query and the trend as the key, perform cross-attention calculation, and generate fused features (which can be represented by z_fused). This allows the current decision to focus on the most relevant historical trends; ultimately, the network can synchronously provide future state predictions and a comprehensive accessibility index.

[0099] In this embodiment, the above method can, in response to a vehicle moving to a wading area, acquire multimodal data of the environment in the wading area. The multimodal data includes modal data collected by multiple sensors in the vehicle at the current moment, and is used to characterize the environmental characteristics of the wading area. Based on the multimodal data, a current state vector corresponding to the wading area is determined, where the current state vector characterizes the physical characteristics of the wading area at the current moment. Based on the current state vector, at least one future state vector of the wading area at at least one future moment is predicted, where the future state vector characterizes the physical characteristics of the wading area at a future moment, which is later than the current moment. Based on the future state vector, a warning message is determined, where the warning message warns of the risks of driving in the wading area. In other words, in this embodiment, the physical characteristics of the wading area are determined based on modal data collected by multiple sensors. Based on the aforementioned multimodal data (i.e., multiple modal data), the physical characteristics of the corresponding wading area at future times are predicted. Corresponding prompt information is generated based on the predicted future state vector. Since the above method predicts the changing trend through multimodal data obtained by multiple sensors, it overcomes the situation where relying solely on a single water depth information or other single perception dimension leads to the inability to perceive the multidimensional physical state of the wading area and the lack of future trend prediction ability. It achieves the goal of continuous, multidimensional, and forward-looking assessment of the wading area, thereby improving the technical effect of risk assessment accuracy during vehicle passage and solving the technical problem of low risk assessment accuracy during vehicle passage.

[0100] The technical solutions of the embodiments of this application will be illustrated below with reference to preferred embodiments.

[0101] Currently, as intelligent driving advances to more advanced levels, vehicles must be able to navigate in adverse weather conditions and complex road conditions. Driving through water is a typical high-risk scenario, and related solutions have significant limitations in terms of perception depth and decision-making intelligence.

[0102] The output of related technologies focuses on the single dimension of water depth. However, whether a vehicle can safely wade through water also depends on the hardness of the underwater substrate (which determines whether the vehicle will get stuck), the state of water surface ripples (which determines whether waves will enter the engine), and the turbidity of the water (which affects perception and drag). Ignoring these factors leads to serious distortions in risk assessment.

[0103] Alternatively, existing fusion technologies often remain at the level of data comparison or decision voting (i.e., "loose fusion"). When a single sensor fails due to environmental interference (e.g., glare from a camera or mirror reflection from a LiDAR sensor on water), system performance drops sharply. The lack of deep fusion at the feature level and physical model level makes it impossible to fundamentally achieve complementary advantages and redundancy between sensors.

[0104] Optionally, many related technologies rely on static judgments based on thresholds or black-box models, or treat turbidity as a simple compensation coefficient. The physical meaning of the entire perception process is unclear, resulting in low reliability. More importantly, these solutions are all "snapshot" assessments of the current moment, completely unable to predict water surface fluctuations, surge generation, or changes in water depth ahead in the next few seconds. This leads to insufficient warning and decision-making time for autonomous driving systems, creating an inherent safety ceiling. The core difficulty in solving these problems lies in establishing a unified theoretical framework to explain the complex interactions between light, electricity, waves, and water, and designing a complete algorithm system capable of real-time operation under limited onboard computing power, fusing multi-source heterogeneous data, simultaneously outputting multiple physical states, and performing time-series predictions. This exceeds the capabilities of traditional signal processing or simple machine learning.

[0105] Optionally, this application aims to address the problem that existing water sensing systems suffer from incomplete information dimensions, failing to simultaneously acquire multiple key physical states such as water depth, turbidity, volatility, and substrate hardness; it also addresses the challenge that existing multi-sensor fusion schemes suffer from shallow layers and low robustness, failing to achieve deep fusion at the feature and physical mechanism levels to cope with partial sensor failures. Furthermore, it addresses the safety issues arising from the lack of physical interpretability and forward-looking predictive capabilities in existing technologies, leading to opaque decision-making criteria and high early warning delays.

[0106] Optionally, this application achieves full-dimensional and interpretable environmental perception: through this application, the system can output four parameters with clear physical meanings in real time: water depth (H), optical turbidity (τ), water surface wave intensity (σ_s), and underwater substrate dielectric constant (ε_r). This upgrades risk assessment from "guessing" based on a single water depth to "diagnosis" based on multiple physical quantities, making the decision-making basis transparent, reliable, and compliant with functional safety requirements.

[0107] Optionally, this application constructs a highly robust deep fusion system: through innovative "dynamic feature weight allocation" and "multi-physics model joint inversion," this application achieves fusion at the feature level and model level. For example, when a camera fails, the system can automatically reduce the visual weight and continue to work by relying on the scattering characteristics of the lidar and the penetration characteristics of the millimeter-wave radar, achieving a redundancy effect of "1+1+1>3" and improving the system reliability under extreme weather conditions.

[0108] Optionally, this application employs a "current-trend" dual-flow time-series prediction network. The system can not only assess current risks but also predict the evolution of water surface conditions (such as swell height) and trafficability indices within the next 2-3 seconds. This provides a valuable early warning window for autonomous driving planners or drivers, realizing a paradigm shift from "passive warning" to "proactive anticipation and defense," and improving the avoidance rate of potential accidents.

[0109] Optionally, by comprehensively assessing the hardness and turbidity of the substrate (which affect driving resistance), the system can recommend the optimal passage route (e.g., choosing a lane with a hard substrate) and suggested speed, effectively reducing the risk of the vehicle getting stuck, skidding, or overloading the power system.

[0110] In this embodiment, a test vehicle equipped with this system is driving at night on a suburban road flooded due to poor drainage. Under the headlights, the camera identifies a large, bright area (water surface) ahead; the lidar shows that the point cloud in this area is sparse but the backscattering intensity is high; the millimeter-wave radar detects a penetrating signal, and the ρ_hv value is low. The dynamic fusion module determines that the information from each sensor is consistent and assigns a balanced weight; joint inversion: the inversion engine solves for θ. The system immediately determines that the water depth is close to the limit, the bottom is slippery, and the water is turbid; the prediction network analyzes the water surface fluctuation sequence of the previous few seconds, and combined with the fact that a truck is approaching from the opposite direction, predicts that when the vehicle and the truck meet in 8 seconds, a transient 0.48-meter surge will be generated due to water disturbance; based on the current assessment and future prediction, the system judges the risk to be extremely high. The system then executes the following: the flooded area is rendered with red highlights on the AR-HUD, and a dynamic surge prediction animation is overlaid; a voice prompt is issued: "The water ahead is too deep and the bottom is slippery. Oncoming vehicles will cause large waves. Please detour immediately." The control sends a "enter wading mode" command to the power domain (such as raising the air intake or adjusting the torque output curve) and requests the navigation system to replan the route.

[0111] In the embodiments of this application, the logic and method for adjusting the confidence weights of each modal feature in real time based on the observation conflicts between sensors are protected, especially the part that uses physical rules (e.g., specular reflection features) for conflict detection; the specific algorithm flow and mathematical model for constructing a unified cost function by using features such as visual, laser scattering, and millimeter-wave polarization as constraints, and jointly solving for water depth, turbidity, volatility, and substrate dielectric constant are protected; and the specific network design consisting of trend flow, current flow, and cross-flow attention fusion modules is protected for the purpose of predicting changes in water surface state and trafficability index.

[0112] In this embodiment, in the dynamic fusion module, besides rule-based weight allocation, a lightweight neural network can be used to learn and output weights. In the temporal prediction network, a gated recurrent unit can be used instead; the cross-current attention mechanism can be replaced by a simple concatenation followed by a fully connected layer for fusion, but performance may decrease. If dual-polarized millimeter-wave radar is not supported, multiple single-polarized radars with different incident angles can be used to indirectly infer similar information through multi-angle observation, but accuracy will be reduced.

[0113] The methods of the embodiments of this application will be further illustrated below.

[0114] Figure 2 This is a flowchart illustrating the overall operation of an optional sensing system according to an embodiment of this application. Figure 2 The overall workflow of the system shown includes the following steps.

[0115] Step S202: Synchronous data acquisition from multiple sensors.

[0116] Optionally, time synchronization can be achieved by using a vehicle-mounted forward-looking camera, lidar, millimeter-wave radar, and ultrasonic array with a unified trigger signal and a sampling frequency of no less than 20 Hz. This ensures that the observations of each sensor on the same water-wading scene are strictly aligned in the time domain, avoiding motion blur or state distortion caused by asynchronous sampling.

[0117] Step S204: Preprocessing and spatiotemporal alignment.

[0118] Optionally, based on calibration parameters (obtained offline by the vehicle calibration system), the point clouds / images of each sensor are unified to the vehicle coordinate system, and spatial resolution matching is completed using interpolation or voxelization methods; using camera frames as a reference, motion compensation is performed using accelerometer data, and motion distortion correction is performed on the lidar point cloud and millimeter-wave radar echo.

[0119] Step S206: Multimodal feature extraction and dynamic fusion.

[0120] Optionally, based on the water surface region mask output by the U-Net segmentation network, the proportion of bright areas in the hue-saturation-brightness space and the texture entropy are calculated to generate a visual turbidity index; the mean echo intensity and pulse broadening of the point cloud in the water surface region are statistically analyzed; the polarization differential reflectance, co-polarization correlation coefficient and Doppler spectral width are extracted; and a rule-driven confidence assessment is performed based on the consistency of observations between sensors.

[0121] Step S208, Physical Model Joint Inversion Engine.

[0122] Optionally, the input is the weighted multimodal features, and the output is a four-dimensional physical state vector (water depth, optical turbidity, water surface wave intensity, and underwater substrate dielectric constant); a forward physical model is constructed; a nonlinear weighted least squares inversion objective function is constructed, and the output is a current state vector with clear physical meaning.

[0123] Step S210, "Current Status Quo-Trend" Time Series Prediction Network.

[0124] Optionally, the input is the physical state obtained from the inversion at the current moment and the historical sequence in the past; a temporal convolutional network is used to model the historical sequence and extract dynamic evolution features; a lightweight MLP is used to concatenately encode the current state and the original sensor features to output the current status representation; and fused features are generated.

[0125] Step S212, Comprehensive decision-making and output.

[0126] Optionally, a dynamic 3D water rendering can be overlaid on the AR-HUD, which can indicate water depth and risk level with color gradients and dynamically play animations predicting future surges; the vehicle's voice system can announce "The water depth ahead is 0.42 meters, the bottom is soft and wet, and oncoming vehicles will cause a 0.48-meter surge, please detour immediately"; and the system can send "Enter wading mode" to the power domain controller, triggering automatic raising of the air intake, limiting torque output, and switching to low-speed cruise mode.

[0127] Figure 3 This is a flowchart of an optional water body multi-attribute joint inversion engine algorithm based on a physical model, according to an embodiment of this application; as follows: Figure 3 As shown, the process of this physical model-based water multi-attribute joint inversion engine algorithm can include the following steps.

[0128] Step S302: Obtain the weighted multimodal feature vector and dynamic confidence weights.

[0129] Optionally, the input is a weighted observation feature vector w_vis, w_lidar, w_radar| from the multimodal feature extraction and dynamic fusion module.

[0130] Step S304: Define the physical state vector θ to be solved.

[0131] Optionally, the target inversion variable is defined as a four-dimensional physical state vector θ=[H,τ,σ_s,ε_r].

[0132] Step S306: Construct a forward physical observation model M(θ).

[0133] Optionally, a deterministic physical relationship model between observations and physical states is constructed for each type of sensor. All models are differentiable, continuous, and analytical functions that support gradient optimization.

[0134] Step S308: Construct a forward physical observation model M(θ) and a weighted nonlinear least squares cost function J(θ).

[0135] Optionally, the error between sensor observations and model predictions can be combined with dynamic confidence weights to construct a global optimization objective function.

[0136] Step S310: Use optimization algorithms such as Levenberg-Marquardt to solve for the optimal physical state estimate.

[0137] Optionally, the LM algorithm is used for iterative solution. The time taken to calculate the Jacobian matrix in each iteration is controlled to be less than 35 milliseconds to meet the real-time requirements of the vehicle, and the optimal physical state estimate θ is output. =argJmin(θ).

[0138] Step S312: Output the optimal physical state estimate.

[0139] Optionally, the output is a four-dimensional physical state vector obtained after optimization and convergence, that is, the optimal physical state estimate θ. It also includes an inversion confidence index.

[0140] Figure 4 This is a flowchart of an optional "current state-of-the-art" dual-stream temporal series prediction fusion network architecture according to an embodiment of this application. Figure 4 As shown, the process of this "current situation-trend" dual-stream time series prediction fusion network architecture may include the following steps.

[0141] Step S402, "Current Status-Trend" dual-stream time series prediction network architecture.

[0142] Optionally, the input data consists of a historical state sequence, the current feature vector F_t, and the vehicle context C_t provided by the preceding module.

[0143] Step S404, Trend Prediction Stream (TCN / GRU).

[0144] Optionally, the historical state sequence is used as input, and long-term dynamic evolution features are extracted through a temporal convolutional network. Each layer of the temporal convolutional network is followed by ReLU activation and layer normalization, and finally outputs trend encoding.

[0145] Step S406, Current Status Flow (MLP).

[0146] Optionally, current environmental cognition can be constructed using a lightweight multilayer perceptron, taking current features and vehicle context as input.

[0147] Step S408, Cross-stream attention fusion module.

[0148] Optionally, the current feature F_t and the vehicle context C_t are used as inputs to construct the current environment cognition through a lightweight multilayer perceptron and output the current status code; the trend code and the current status code are input into the cross-flow attention mechanism, the current status code is used as the query vector, and the trend code is used as the key vector and value vector, and the values ​​are weighted and aggregated to generate fused features.

[0149] Step S410, regression prediction.

[0150] Optionally, the fused features are input into the dual-task output head, which includes a future state prediction head and a mobility index prediction head, and outputs a mobility index.

[0151] Step S412, evaluate the decision.

[0152] Optionally, based on the above-mentioned tiered collaborative decision-making of the wading index, the AR-HUD can overlay dynamic three-dimensional water body rendering, predict wave trajectory animation, and display real-time wading index; it can also send a "enter wading mode" command to the power domain: raise the air intake, limit torque, and switch to low-speed cruise.

[0153] Figure 5 This is a schematic diagram of an optional system decision output and human-computer interaction interface according to an embodiment of this application. Figure 5 As shown, the system's decision output and human-machine interface include a top warning bar, a middle AR view, and a bottom decision area. The top warning bar displays the road conditions ahead and the risk level; the middle AR view displays lane water depth gauges, turbidity, and visible fluctuations; and the bottom decision area displays perception details, underwater details, decision suggestions, predictive prompts, and vehicle modes. It should be noted that the text content in the attached diagram can be changed according to actual circumstances.

[0154] In this embodiment, the above method can, in response to a vehicle moving to a wading area, acquire multimodal data of the environment in the wading area. The multimodal data includes modal data collected by multiple sensors in the vehicle at the current moment, and is used to characterize the environmental characteristics of the wading area. Based on the multimodal data, a current state vector corresponding to the wading area is determined, where the current state vector characterizes the physical characteristics of the wading area at the current moment. Based on the current state vector, at least one future state vector of the wading area at at least one future moment is predicted, where the future state vector characterizes the physical characteristics of the wading area at a future moment, which is later than the current moment. Based on the future state vector, a warning message is determined, where the warning message warns of the risks of driving in the wading area. In other words, in this embodiment, the physical characteristics of the wading area are determined based on modal data collected by multiple sensors. Based on the aforementioned multimodal data (i.e., multiple modal data), the physical characteristics of the corresponding wading area at future times are predicted. Corresponding prompt information is generated based on the predicted future state vector. Since the above method predicts the changing trend through multimodal data obtained by multiple sensors, it overcomes the situation where relying solely on a single water depth information or other single perception dimension leads to the inability to perceive the multidimensional physical state of the wading area and the lack of future trend prediction ability. It achieves the goal of continuous, multidimensional, and forward-looking assessment of the wading area, thereby improving the technical effect of risk assessment accuracy during vehicle passage and solving the technical problem of low risk assessment accuracy during vehicle passage.

[0155] Figure 6 This is a schematic diagram of a device for determining prompt information in a vehicle according to an embodiment of this application. Figure 6 As shown, the device for determining the prompt information in the vehicle may include: an acquisition unit 602, used to acquire multimodal data of the environment of the wading area in response to the vehicle moving to the wading area, wherein the multimodal data includes modal data collected by multiple sensors in the vehicle at the current moment, and the multimodal data is used to characterize the environmental characteristics of the wading area.

[0156] The processing unit 604 is used to determine the current state vector corresponding to the water-contaminated area based on multimodal data, wherein the current state vector is used to characterize the physical characteristics of the water-contaminated area at the current moment.

[0157] The prediction unit 606 is used to predict at least one future state vector of the water-crossing area at at least one future time based on the current state vector, wherein the future state vector is used to characterize the physical characteristics of the water-crossing area at a future time, and the future time is later than the current time.

[0158] The determining unit 608 is used to determine the prompt information based on the future state vector, wherein the prompt information is used to indicate the risk of the vehicle driving in the wading area.

[0159] In this embodiment, the acquisition unit acquires multimodal data of the environment of the wading area in response to the vehicle moving to the wading area. This multimodal data includes modal data collected by multiple sensors in the vehicle at the current moment, and is used to characterize the environmental characteristics of the wading area. The processing unit determines the current state vector corresponding to the wading area based on the multimodal data. This current state vector characterizes the physical characteristics of the wading area at the current moment. The prediction unit predicts at least one future state vector of the wading area at at least one future moment based on the current state vector. This future state vector characterizes the physical characteristics of the wading area at a future moment, which is later than the current moment. The determination unit determines a warning message based on the future state vector. This warning message indicates the risk of the vehicle driving in the wading area, thereby improving the accuracy of risk assessment during vehicle passage and solving the technical problem of low risk assessment accuracy during vehicle passage.

[0160] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0165] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for determining prompt information in a vehicle, characterized in that, include: In response to the vehicle moving to a wading area, multimodal data of the environment in the wading area is acquired, wherein the multimodal data includes modal data collected by multiple sensors in the vehicle at the current moment, and the multimodal data is used to characterize the environmental characteristics of the wading area; Based on the multimodal data, the current state vector corresponding to the water-contaminated area is determined, wherein the current state vector is used to characterize the physical characteristics of the water-contaminated area at the current moment; Based on the current state vector, at least one future state vector of the water-crossing area at at least one future time is predicted, wherein the future state vector is used to characterize the physical characteristics of the water-crossing area at the future time, and the future time is later than the current time. Based on the future state vector, a warning message is determined, wherein the warning message is used to warn the vehicle of the risks of driving in the flooded area.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the historical state vector sequence corresponding to the water-contacting area, wherein the historical state vector sequence includes at least one historical state vector corresponding to a historical time period, and is used to characterize the changing trend of the physical characteristics of the water-contacting area during the historical time period, wherein the historical time period is earlier than the current time. The step of predicting at least one future state vector of the water-affected area at at least one future time based on the current state vector includes: Based on the current state vector, a current status code is determined, and based on the historical state vector sequence, a trend code is determined, wherein the current status code is used to characterize the instantaneous state of the water-contaminated area at the current moment, and the trend code is used to characterize the historical state change trend of the water-contaminated area. Based on the current state encoding and the trend encoding, the future state vector is predicted.

3. The method according to claim 2, characterized in that, The step of determining the current state vector corresponding to the water-affected area based on the multimodal data includes: The multimodal data is identified to obtain the vehicle's driving condition information; Determine weighted data that matches the driving condition information, wherein the weighted data is used to characterize the degree of influence of different modal data in the multimodal data on the physical characteristics; The weighted data is used to transform the multimodal data to obtain the current state vector.

4. The method according to claim 3, characterized in that, The step of transforming the multimodal data using the weight data to obtain the current state vector includes: Feature extraction is performed on the multimodal data to obtain multiple initial feature data; The initial feature data is processed by calling the physical forward model to obtain an initial state vector, wherein the physical forward model is used to characterize the correlation between the initial feature data and the standard state vector; The differences between the multiple initial state vectors and the multiple standard state vectors are determined respectively to obtain multiple differences; Using the weight data, multiple differences are transformed to obtain an objective function, wherein the objective function is used to characterize the degree of error between the initial state vector and the standard state vector; The objective function is nonlinearly adjusted to obtain the current state vector.

5. The method according to claim 4, characterized in that, The process of obtaining the current state code based on the current state vector includes: The current state vector and the initial state vector are encoded to obtain the current state code.

6. The method according to claim 2, characterized in that, The process of predicting the future state vector based on the current state encoding and the trend encoding includes: The current status code and the trend code are fused to obtain a fused feature; The fused features are identified to determine the future state vector; The method further includes: The fusion features are identified to determine the traversability index of the water crossing area, wherein the traversability index is used to characterize the likelihood that the vehicle can safely pass through the water crossing area.

7. A device for determining prompt information in a vehicle, characterized in that, include: The acquisition unit is configured to acquire multimodal data of the environment of the wading area in response to the vehicle moving to the wading area, wherein the multimodal data includes modal data collected by multiple sensors in the vehicle at the current moment, and the multimodal data is used to characterize the environmental characteristics of the wading area. The processing unit is configured to determine the current state vector corresponding to the water-contaminated area based on the multimodal data, wherein the current state vector is used to characterize the physical characteristics of the water-contaminated area at the current moment; The prediction unit is configured to predict at least one future state vector of the water-crossing area at at least one future time based on the current state vector, wherein the future state vector is used to characterize the physical characteristics of the water-crossing area at the future time, and the future time is later than the current time. A determining unit is configured to determine a warning message based on the future state vector, wherein the warning message is used to warn the vehicle of the risks of driving in the wading area.

8. A vehicle, characterized in that, Used to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 6.

10. A processor, characterized in that, The processor is used to run a program, wherein the program is executed by the processor to perform the method according to any one of claims 1 to 6.

Images