Vehicle control method and device, storage medium and program product

Abstract

The invention relates to a vehicle control method and device, a storage medium and a program product. The control method comprises the steps that track information of a moving object on a current driving road of a vehicle and position information of a water accumulation area are determined; based on the position information of the ponding area, first radar data of the ponding area collected by a first radar installed on the vehicle and second radar data of the ponding area collected by a second radar are obtained; the first radar and the second radar are different in type; determining depth information of the ponding area based on the first radar data of the ponding area and the second radar data of the ponding area; under the condition of determining that the vehicle can safely pass through the ponding area based on the depth information of the ponding area, determining the risk level of the splashing risk when the moving object passes through the road based on the running speed of the vehicle and the track information of the moving object; and determining a control strategy of the vehicle based on the risk level of the splashing risk.

Description

Technical Field

[0001] The present invention relates to the field of automobile technology, and in particular to a vehicle control method, device, storage medium and program product. Background Art

[0002] Due to the unpredictable weather, especially the problem of road waterlogging caused by rainfall, on the one hand, vehicles may drive into too deep water areas, resulting in serious problems such as water ingress to the engine and electrical system failure. On the other hand, when vehicles pass through the waterlogged road surface, splashing water may affect moving objects on the road, such as pedestrians and cyclists. In the related technology, a single type of sensor is usually relied on to detect the information of the waterlogged area. However, due to the limitations of a single sensor, the accuracy of the information obtained from the waterlogged area is problematic, and it is impossible to effectively protect moving objects from safely passing through the splashing area when the vehicle can safely pass through the waterlogged area. Summary of the invention

[0003] One of the purposes of the present invention is to provide a vehicle control method, device, storage medium and program product, so as to effectively protect a moving object from safely passing through a splashing area while the vehicle can safely pass through a water accumulation area.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] A vehicle control method, the control method comprising: determining trajectory information of a moving object and position information of a water accumulation area on a road on which the vehicle is currently traveling; acquiring first radar data of the water accumulation area collected by a first radar installed on the vehicle and second radar data of the water accumulation area collected by a second radar based on the position information of the water accumulation area; the first radar and the second radar are of different types; determining depth information of the water accumulation area based on the first radar data of the water accumulation area and the second radar data of the water accumulation area; in a case where it is determined based on the depth information of the water accumulation area that the vehicle can safely pass through the water accumulation area, determining the risk level of water splashing risk of the moving object when passing through the road based on the vehicle's driving speed and the trajectory information of the moving object; and determining a control strategy for the vehicle based on the risk level of water splashing risk.

[0006] According to the above-mentioned technical means, firstly, the data collected by at least one device installed on the vehicle can make up for the shortcomings of a single device, and improve the accuracy of the location information of the water accumulation area and the trajectory information of the moving object. Then, the first radar data and the second radar data are determined based on the location information of the water accumulation area with improved accuracy, thereby improving the accuracy of the calculation of the depth information of the water accumulation area determined based on the first radar data and the second radar data. Finally, based on the depth information of the water accumulation area with improved accuracy, the risk level of the moving object when passing through the road and the corresponding vehicle control strategy are determined, which can effectively protect the moving object from safely passing through the water accumulation area when the vehicle can safely pass through the water accumulation area, thereby improving the level of road traffic safety and reducing the occurrence of road accidents.

[0007] In some embodiments, the control method includes: when it is determined that the vehicle cannot safely pass through the flooded area based on the depth information of the flooded area, determining the control strategy of the vehicle as: outputting warning information through the display module of the vehicle and / or controlling the vehicle to perform braking operations.

[0008] According to the above-mentioned technical means, it is determined that the vehicle cannot safely pass through the flooded area based on the depth information of the flooded area with improved accuracy. In this way, when the flooded area is too deep, an emergency alarm prompt or braking measures can be made to solve serious problems such as water ingress into the engine and electrical system failure when the vehicle passes through the flooded area, thereby protecting the lives of the driver and passengers in the vehicle.

[0009] In some embodiments, depth information of the waterlogged area is determined based on first radar data of the waterlogged area and second radar data of the waterlogged area, including: determining first depth information of the waterlogged area based on the first radar data; determining second depth information of the waterlogged area based on the second radar data; determining depth information of the waterlogged area based on the first depth information and the second depth information.

[0010] According to the above-mentioned technical means, the first radar data and the second radar data determined by the data collected by the first radar and the second radar installed on the vehicle can make up for the shortcomings of a single radar device. In this way, the depth information of the water accumulation area determined based on the first radar data and the second radar data improves the calculation accuracy of the depth information of the water accumulation area.

[0011] In some embodiments, based on the vehicle's driving speed and the moving object's trajectory information, the risk level of water splashing risk for the moving object when passing through the road is determined, including: determining the splashing area triggered when the vehicle passes through the water accumulation area based on the depth of the water accumulation area and the vehicle's driving speed; determining the first time when the moving object enters the splashing area based on the moving object's trajectory information and the splashing area; determining the second time when the vehicle enters the splashing area based on the vehicle's driving speed and the splashing area; and determining the risk level of water splashing risk for the moving object when passing through the road based on the first time and the second time.

[0012] According to the above-mentioned technical means, based on the depth information of the water accumulation area with improved accuracy, the calculation accuracy of the splashing area when the vehicle passes through the water accumulation area can be improved, so as to determine the risk level of the mobile object when passing through the road based on the splashing area with improved accuracy. This improves the accuracy of determining the risk level and can effectively protect the mobile object from safely passing through the splashing area.

[0013] In some embodiments, based on the first time and the second time, the risk level of the risk of water splashing when the moving object passes through the road is determined, including: based on the first time, the second time and the first time threshold, determining that the risk level of the risk of water splashing when the moving object passes through the road is high risk; based on the first time, the second time and the second time threshold, determining that the risk level of the risk of water splashing when the moving object passes through the road is medium risk; the second time threshold is greater than the first time threshold; based on the first time, the second time and the third time threshold, determining that the risk level of the risk of water splashing when the moving object passes through the road is low risk; the third time threshold is greater than or equal to the second time threshold.

[0014] According to the above technical means, the relationship between the first time and the second time under different risk levels is explained, so that the risk level of splashing risk for the moving object corresponding to the flooded area when the vehicle passes through the flooded area can be determined based on the actual first time and second time, so that the control strategy corresponding to the risk level can be determined to effectively protect the moving object to safely pass through the splashing area.

[0015] In some embodiments, based on the risk level of water splashing risk, a control strategy of the vehicle is determined, including: when the risk level of water splashing risk is high risk, the control strategy of the vehicle is determined as: sending a warning message to the driver of the vehicle through the display module and / or voice module of the vehicle to remind the driver to slow down or brake; when the risk level of water splashing risk is medium risk, the control strategy of the vehicle is determined as: sending a warning message to the driver of the vehicle through the display module and / or voice module of the vehicle to remind the driver to slow down; when the risk level of water splashing risk is low risk, the control strategy of the vehicle is determined as: controlling the vehicle not to send a warning message, so that the vehicle drives according to the current driving state.

[0016] According to the above technical means, the control strategies of vehicles under different risk levels are explained, so that the control strategy corresponding to the risk level can be determined based on the risk level of splashing risk of mobile objects corresponding to the water accumulation area on the actual driving road of the vehicle, so as to effectively protect the mobile objects to safely pass through the splashing area.

[0017] In some embodiments, the driving mode includes automatic driving, and the control strategy of the vehicle is determined based on the risk level of water splash risk, including: when the risk level of water splash risk is high risk and the driving mode is automatic driving, the control strategy of the vehicle is determined to be: controlling the vehicle to perform emergency braking or emergency deceleration; when the risk level of water splash risk is medium risk and the driving mode is automatic driving, the control strategy of the vehicle is determined to be: controlling the vehicle to perform a gentle deceleration operation; when the risk level of water splash risk is low risk and the driving mode is automatic driving, the control strategy of the vehicle is determined to be: controlling the vehicle to maintain the current speed.

[0018] According to the above technical means, the control strategies of vehicles under different risk levels are explained, so that the control strategy corresponding to the risk level can be determined based on the risk level of splashing risk of mobile objects corresponding to the water accumulation area on the actual driving road of the vehicle, so as to effectively protect the mobile objects to safely pass through the splashing area.

[0019] In some embodiments, determining location information of a water accumulation area on a road on which a vehicle is currently traveling includes: determining first feature information in each image data under a bird's-eye view of the road based on image data collected by multiple cameras; determining second feature information in the third radar data under a bird's-eye view based on third radar data collected by a third radar installed on the vehicle; the first radar, the second radar, and the third radar are of different types; and determining location information of a water accumulation area on the road on which the vehicle is currently traveling based on the first feature information and the second feature information.

[0020] According to the above technical means, firstly, the location information of the waterlogged area is determined by collecting data from a third radar and at least one camera installed on the vehicle, which can make up for the shortcomings of a single device and improve the accuracy of calculating the location information of the waterlogged area.

[0021] In some embodiments, the control method includes: determining the location information of the water accumulation area determined based on the third radar data; correcting the location information of the water accumulation area based on the radar data collected by the first radar, the radar data collected by the second radar and the third radar data; wherein, based on the location information of the water accumulation area, first radar data of the water accumulation area collected by the first radar installed on the vehicle and second radar data of the water accumulation area collected by the second radar are obtained, including: based on the corrected location information of the water accumulation area, obtaining the first radar data of the water accumulation area collected by the first radar installed on the vehicle and the second radar data of the water accumulation area collected by the second radar.

[0022] According to the above-mentioned technical means, firstly, the location information of the water accumulation area determined by the third radar is corrected by using the data collected by the third radar, the first radar and the second radar installed on the vehicle, which can make up for the shortcomings of a single device and improve the accuracy of calculating the location information of the water accumulation area.

[0023] A control device for a vehicle, wherein a first radar and a second radar are arranged on the vehicle; the control device comprises: a first acquisition module, used to acquire radar data collected by the first radar installed on the vehicle and radar data collected by the second radar installed on the vehicle; a first determination module, used to determine the trajectory information of a moving object on the road on which the vehicle is currently traveling and the position information of a water accumulation area; a second acquisition module, used to acquire first radar data of the water accumulation area collected by the first radar based on the position information of the water accumulation area and the radar data collected by the first radar, and to acquire second radar data of the water accumulation area collected by the second radar based on the position information of the water accumulation area and the radar data collected by the second radar; the first radar and the second radar are of different types; a second determination module, used to determine depth information of the water accumulation area based on the first radar data of the water accumulation area and the second radar data of the water accumulation area; a third determination module, used to determine the risk level of water splashing risk of the moving object when passing through the road based on the driving speed of the vehicle and the trajectory information of the moving object, when it is determined that the vehicle can safely pass through the water accumulation area based on the depth information of the water accumulation area; a fourth determination module, used to determine the control strategy of the vehicle based on the risk level of water splashing risk.

[0024] A computer-readable storage medium stores a computer program, which, when executed by a control device of a vehicle, implements part or all of the steps in any of the above methods.

[0025] A computer program product includes a computer program or instructions. When the computer program or instructions are executed by a control device of a vehicle, some or all of the steps in the method described in the claim are implemented.

[0026] Beneficial effects of the present invention:

[0027] (1) The data collected by the present invention through at least one device installed on the vehicle can complement the deficiencies of a single device and improve the accuracy of the location information of the flooded area and the trajectory information of the moving object;

[0028] (2) The present invention determines the first radar data and the second radar data based on the location information of the waterlogged area with improved accuracy, thereby improving the accuracy of calculating the depth information of the waterlogged area determined based on the first radar data and the second radar data;

[0029] (3) The present invention determines the risk level of water splashing when a moving object passes through a road based on the depth information of the water accumulation area with improved accuracy, and the corresponding vehicle control strategy. This can effectively protect the moving object from passing through the water splashing area safely when the vehicle can pass through the water accumulation area safely, thereby improving the level of road traffic safety and reducing the occurrence of road accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of an implementation flow of a vehicle control method provided by the present invention;

[0031] Figure 2 A schematic diagram of a vehicle control system based on multi-sensor fusion provided by the present invention;

[0032] Figure 3 A schematic diagram of the implementation flow of a two-dimensional occupancy grid algorithm provided by the present invention;

[0033] Figure 4 The present invention provides a schematic diagram of the structure of a vehicle control device. DETAILED DESCRIPTION

[0034] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0035] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0036] This embodiment proposes a vehicle control method, such as Figure 1 As shown, the method may include steps S101 to S105:

[0037] Step S101: determining the trajectory information of the moving object on the road the vehicle is currently traveling on and the location information of the waterlogged area;

[0038] Here, the moving object refers to an object that moves on the road and whose spatial position changes accordingly with time, such as pedestrians, motorcyclists, animals, etc. The trajectory information of the moving object refers to the route information of the moving object.

[0039] In some embodiments, the mobile object may be a vulnerable road user (Vulnerable Road User, VRU), where VRU generally refers to individuals who use transportation vehicles on the road but have less protection, such as pedestrians, cyclists, motorcyclists, people driving electric scooters or skateboards, etc. These groups are more likely to be injured in traffic accidents than motor vehicle drivers.

[0040] In some implementations, the trajectory information of moving objects and the location information of water accumulation areas on the road the vehicle is currently traveling on may be determined through data collected by a device installed on the vehicle.

[0041] Step S102: acquiring first radar data of the water accumulation area collected by a first radar installed on the vehicle and second radar data of the water accumulation area collected by a second radar based on the position information of the water accumulation area; the first radar and the second radar are of different types;

[0042] Here, the first radar and the second radar refer to radars that detect information such as the position, speed, and direction of an object by transmitting and receiving signals, wherein the detection distance of the first radar is shorter, and the detection distance of the second radar is longer than that of the first radar. For example, the first radar may be an ultrasonic radar, and the second radar may be a millimeter wave radar.

[0043] In some embodiments, the acquisition module sends radar data collected by a first radar and a second radar installed on the vehicle to a control device of the vehicle, and the control device of the vehicle filters out the first radar data of the water accumulation area from the radar data of the first radar, and filters out the second radar data of the water accumulation area from the radar data of the second radar.

[0044] Step S103: determining depth information of the waterlogged area based on the first radar data of the waterlogged area and the second radar data of the waterlogged area;

[0045] In some embodiments, first, the first depth information of the waterlogged area can be determined by detecting the time difference between a signal reflected by the water surface and a signal that partially penetrates the water surface in the waterlogged area using a first radar, and the second depth information of the waterlogged area can be determined by detecting the time difference between a signal reflected by the water surface and a signal that partially penetrates the water surface in the waterlogged area using a second radar; then, the first depth information and the second depth information are weightedly calculated to obtain the depth information of the waterlogged area.

[0046] Step S104: when it is determined based on the depth information of the waterlogged area that the vehicle can safely pass through the waterlogged area, the risk level of the risk of water splashing when the moving object passes through the road is determined based on the driving speed of the vehicle and the trajectory information of the moving object;

[0047] Here, the risk level is used to characterize the degree of risk of water splashing when a vehicle passes through a flooded area, and may include high risk, low risk, and medium risk.

[0048] In some embodiments, a safety threshold of the vehicle is obtained, and based on the safety threshold of the vehicle and the depth information of the water accumulation area, it is determined whether the vehicle can safely pass through the water accumulation area, wherein when the safety threshold of the vehicle is greater than the depth information of the water accumulation area, it is determined that the vehicle can safely pass through the water accumulation area. The safety threshold of the vehicle may be determined based on the wading performance of the vehicle and the maximum safe wading depth recommended by the vehicle manufacturer, for example, for an ordinary passenger car, the safety threshold may be 40 centimeters.

[0049] In some embodiments, first, the splashing area when the vehicle passes through the water accumulation area can be determined based on the vehicle's driving speed and the depth information of the water accumulation area; then, when the trajectory information of the moving object indicates that the moving object will pass through the splashing area, the first time when the moving object enters the splashing area and the second time when the vehicle enters the splashing area are determined respectively; finally, the risk level of the risk of splashing when the moving object passes through the road is determined based on the first time and the second time.

[0050] Step S105: Determine a control strategy for the vehicle based on the risk level of water splashing risk.

[0051] In this embodiment, firstly, the data collected by at least one device installed on the vehicle can make up for the shortcomings of a single device, and improve the accuracy of the location information of the water accumulation area and the trajectory information of the moving object. Then, the first radar data and the second radar data are determined based on the location information of the water accumulation area with improved accuracy, thereby improving the accuracy of the calculation of the depth information of the water accumulation area determined based on the first radar data and the second radar data. Finally, based on the depth information of the water accumulation area with improved accuracy, the risk level of the moving object when passing through the road and the corresponding vehicle control strategy are determined, which can effectively protect the moving object from safely passing through the water accumulation area when the vehicle can safely pass through the water accumulation area, thereby improving the level of road traffic safety and reducing the occurrence of road accidents.

[0052] In some embodiments, the above steps S104 and S105 may also be replaced by step S106:

[0053] Step S106: When it is determined based on the depth information of the flooded area that the vehicle cannot safely pass through the flooded area, a control strategy for the vehicle is determined as follows: outputting a warning message through the display module and / or voice module of the vehicle and / or controlling the vehicle to perform a braking operation.

[0054] In some embodiments, a safety threshold of the vehicle is obtained, and based on the safety threshold of the vehicle and the depth information of the flooded area, it is determined whether the vehicle can safely pass through the flooded area, wherein if the safety threshold of the vehicle is less than the depth information of the flooded area, it is determined that the vehicle cannot safely pass through the flooded area.

[0055] It should be noted that when the vehicle's safety threshold is less than the depth information of the flooded area, it means that when the vehicle passes through the flooded area, serious problems such as water entering the vehicle engine and electrical system failure may occur, threatening the lives of the driver and passengers in the vehicle. Therefore, it is necessary to issue an early warning to the driver of the vehicle.

[0056] In some embodiments, when it is determined that the vehicle cannot safely pass through the flooded area based on the depth information of the flooded area, the vehicle control strategy is determined as: sending a warning message to the driver of the vehicle through the vehicle's display module and / or voice module to remind the driver to brake.

[0057] In some embodiments, when the driving mode of the vehicle is automatic driving, and it is determined based on the depth information of the water accumulation area that the vehicle cannot safely pass through the water accumulation area, the control strategy of the vehicle is determined as: sending a warning message to the driver of the vehicle through the display module and / or voice module of the vehicle, and controlling the vehicle to perform a braking operation. The warning message is sent to the driver of the vehicle through the display module and / or voice module of the vehicle in order to make the driver understand the automatic driving strategy.

[0058] The above-mentioned technical means, based on the depth information of the flooded area with improved accuracy, determines that the vehicle cannot safely pass through the flooded area. In this way, when the flooded area is too deep, emergency alarm prompts or braking measures can be issued to solve serious problems such as water ingress into the engine and electrical system failure when the vehicle passes through the flooded area, thereby protecting the lives of the driver and passengers in the vehicle.

[0059] In some embodiments, the above step S103, determining the depth information of the water accumulation area based on the first radar data of the water accumulation area and the second radar data of the water accumulation area, may include steps S1031 to S1033:

[0060] Step S1031: determining first depth information of a waterlogged area based on the first radar data;

[0061] The first depth information refers to formula (1):

[0062]

[0063] Among them, D u represents the first depth information, v represents the propagation speed of sound waves in the air, and △t is the round-trip time difference of the signal.

[0064] Step S1032: determining second depth information of the waterlogged area based on the second radar data;

[0065] The second depth information refers to formula (2):

[0066]

[0067] Among them, D m represents the second depth information, c represents the propagation speed of electromagnetic waves in the air, and △t is the round-trip time difference of the signal.

[0068] Step S1033: Determine the depth information of the water accumulation area based on the first depth information and the second depth information.

[0069] The depth information of the waterlogged area can be found in formula (3):

[0070] D fused =(1-w)D m +wD u (3);

[0071] Among them, D fused Indicates the depth information of the waterlogged area, D u Indicates the first depth information, D m represents the second depth information, and w represents the weight coefficient of the first radar.

[0072] Furthermore, the weight calculation method is shown in formula (4):

[0073]

[0074] Among them, d u Indicates the distance between the first radar and the flooded area, and α is an adjustment parameter used to control the influence of distance on the weight. The larger the α value, the more significant the influence of distance on the weight, and the smaller the α value, the less influence of distance on the weight.

[0075] The above-mentioned technical means, through the first radar data and the second radar data collected by the first radar and the second radar installed on the vehicle, can make up for the shortcomings of a single radar device. In this way, the depth information of the water accumulation area determined based on the first radar data and the second radar data improves the calculation accuracy of the depth information of the water accumulation area.

[0076] In some embodiments, the above step S104, based on the vehicle's driving speed and the trajectory information of the moving object, determines the risk level of water splashing risk when the moving object passes through the road, and may include steps S1041 to S1044:

[0077] Step S1041: determining a splashing area triggered when the vehicle passes through the water accumulation area based on the depth of the water accumulation area and the driving speed of the vehicle;

[0078] Here, the splashing area refers to the area where water splashes may be generated by tire squeezing when a vehicle passes through a flooded area.

[0079] In some embodiments, the splash area may be a fan-shaped area extending in the direction of travel of the vehicle. The calculation method of the splash area is shown in formula (5):

[0080]

[0081] Among them, R splash represents the splash area corresponding to the water accumulation area, β represents the empirical coefficient, v represents the vehicle's speed, and D fused represents the depth information of the water accumulation area, and k represents the friction coefficient between the vehicle's tires and the road surface.

[0082] Step S1042: determining the first time when the moving object enters the splash area based on the trajectory information of the moving object and the splash area;

[0083] In some embodiments, it is possible to first determine whether the trajectory information of the moving object intersects with the splash area, so as to determine whether the moving object will enter the splash area, and then determine the first time when the moving object enters the splash area based on the moving speed and moving direction of the moving object.

[0084] Step S1043: determining a second time when the vehicle enters the splash area based on the vehicle's travel speed and the splash area;

[0085] In some embodiments, first, the driving route and driving speed of the vehicle are obtained; then, based on the driving route and driving speed of the vehicle, the second time when the vehicle enters the splash zone can be determined. The driving speed of the vehicle can be detected by a speed sensor installed on the vehicle. The driving route of the vehicle can be determined based on the destination address of the vehicle, or can be predicted by a deep learning algorithm based on the driving direction and driving speed of the vehicle.

[0086] Step S1044: Based on the first time and the second time, determine the risk level of water splashing risk when the moving object passes through the road.

[0087] Here, the risk level of water splashing risk when the moving object passes through the road can be determined through the relationship between the first time and the second time.

[0088] The above-mentioned technical means, based on the depth information of the water accumulation area with improved accuracy, can improve the calculation accuracy of the splashing area when the vehicle passes through the water accumulation area, thereby determining the risk level of the mobile object's splashing risk when passing through the road based on the splashing area with improved accuracy. This improves the accuracy of determining the risk level and can effectively protect the mobile object from safely passing through the splashing area.

[0089] In some embodiments, the above step S1044, based on the first time and the second time, determines the risk level of water splashing risk when the moving object passes through the road, and may include steps S1144 to S1146:

[0090] Step S1144: Based on the first time, the second time and the first time threshold, determining that the risk level of water splashing risk when the moving object passes through the road is high risk;

[0091] In some embodiments, when the absolute value of the difference between the first time and the second time is less than the first time threshold, it can be determined that the vehicle and the moving object enter the splashing area at the same time, and therefore the risk level of the splashing risk when the moving object passes through the road is high.

[0092] Step S1145: Based on the first time, the second time and the second time threshold, determining that the risk level of water splashing risk when the moving object passes through the road is medium risk; the second time threshold is greater than the first time threshold;

[0093] In some embodiments, when the first time is less than the second time, and the absolute value of the difference between the first time and the second time is greater than the first time threshold and less than the second time threshold, it can be determined that although the vehicle and the moving object did not enter the splashing area at the same time, the time difference between their entry was close, and therefore the risk level of the splashing risk when the moving object passes through the road is medium risk.

[0094] Step S1146: Based on the first time, the second time and the third time threshold, determine that the risk level of water splashing risk when the moving object passes through the road is low risk; the third time threshold is greater than or equal to the second time threshold.

[0095] In some implementations, when the absolute value of the difference between the first time and the second time is greater than the third time threshold, it can be determined that the risk level of water splashing risk when the moving object passes through the road is low risk.

[0096] In other embodiments, when it can be determined based on the state information of the moving object that the moving object has an obvious water splash avoidance area, the risk level of the water splash risk of the moving object when passing through the road is determined to be low risk.

[0097] The above-mentioned technical means illustrate the relationship between the first time and the second time under different risk levels. In this way, based on the actual first time and second time, the risk level of splashing risk for the moving object corresponding to the flooded area when the vehicle passes through the flooded area can be determined, thereby determining the control strategy corresponding to the risk level to effectively protect the moving object from safely passing through the splashing area.

[0098] In some embodiments, the above step S105, determining the control strategy of the vehicle based on the risk level of water splash risk, may include steps S1051 to S1053:

[0099] Step S1051: when the risk level of water splashing risk is high, determining the control strategy of the vehicle as follows: sending a warning message to the driver of the vehicle through the display module and / or voice module of the vehicle to remind the driver to decelerate or brake;

[0100] Here, the display module can be a bridge for information interaction between the user and the vehicle, used to display various information of the vehicle and provide the driver with necessary driving references and vehicle status feedback. For example, the display module can be an on-board display or a head-up display (HUD).

[0101] In some embodiments, the warning information may be sent to the driver through a display module of the vehicle, may be sent to the driver through a voice module of the vehicle, or may be sent to the driver through a display module and a voice module of the vehicle.

[0102] In some embodiments, the display module can also display in real time the position of the moving object, the movement trajectory of the moving object, the information of the water accumulation area, the water splashing area and other information, so that the driver can understand the current water splashing risk, wherein the information of the water accumulation area may include the position information of the water accumulation area, the depth information of the water accumulation area and the depth information of the water accumulation area, etc.

[0103] It should be noted that when the risk level of water splashing is high, it means that water is about to splash onto the moving object and the driver needs to take emergency avoidance measures. Therefore, when the driver receives the warning information, he should take emergency braking or emergency deceleration measures to avoid splashing onto the moving object.

[0104] Step S1052: when the risk level of the splash risk is medium risk, determining the control strategy of the vehicle as follows: sending a warning message to the driver of the vehicle through the display module and / or voice module of the vehicle to remind the driver to decelerate;

[0105] It should be noted that when the risk level of water splashing is medium, if the vehicle maintains its current driving state, water will splash onto moving objects in the future, requiring the driver to take avoidance measures. Therefore, when the driver receives the warning information, he should take measures to decelerate gently to avoid splashing onto moving objects in the future.

[0106] Step S1053: When the risk level of the water splashing risk is low risk, the control strategy of the vehicle is determined as: controlling the vehicle not to issue a warning message so that the vehicle drives according to the current driving state.

[0107] Here, driving status refers to the specific condition or mode of a vehicle when it is driving on the road. It can be classified and described according to multiple dimensions such as the vehicle's driving power, driving speed, driving direction, and driving stability.

[0108] In some embodiments, when the risk level of water splash risk is low, there is no need to send a warning message to the driver, so the vehicle can continue to travel in the current driving state.

[0109] In other embodiments, although the risk level of water splashing risk of the moving object is low, there may still be some potential water splashing areas, in which case the potential water splashing areas can be displayed by the display module. It should be noted that, under this risk level, the water splashing risk when the moving object passes through the potential water splashing area is still low, but by displaying the potential water splashing area by the display module, the driver can dynamically adjust the driving state of the vehicle at any time according to the displayed information.

[0110] The above-mentioned technical means illustrate the control strategies of vehicles under different risk levels. In this way, based on the risk level of splashing risk of mobile objects corresponding to the water accumulation area on the actual driving road of the vehicle, the control strategy corresponding to the risk level can be determined to effectively protect the mobile objects to safely pass through the splashing area.

[0111] In some embodiments, the driving mode includes automatic driving, and the above step S105, based on the risk level of water splashing risk, determines the control strategy of the vehicle, which may include steps S1054 to S1056:

[0112] Step S1054: when the risk level of the splash risk is high and the driving mode is automatic driving, determining the control strategy of the vehicle as: controlling the vehicle to perform an emergency braking or emergency deceleration operation;

[0113] Here, autonomous driving refers to the ability of a vehicle to autonomously perform driving tasks without direct human driver control. When the vehicle is autonomously driving, it uses onboard sensors to sense the vehicle's surroundings and controls the vehicle's steering and speed based on the road, vehicle position and obstacle information it senses, so that the vehicle can drive safely and reliably on the road.

[0114] In some embodiments, when the driving mode of the vehicle is automatic driving, a warning message can also be sent to the driver of the vehicle through the vehicle display module and / or voice module to enable the driver to understand the current automatic driving control strategy.

[0115] It should be noted that when the risk level of water splashing is high, it means that water is about to splash onto the moving object and emergency avoidance measures are required. Therefore, the vehicle should be controlled to take emergency braking or emergency deceleration measures to avoid splashing onto the moving object.

[0116] Step S1055: when the risk level of the splash risk is medium risk and the driving mode is automatic driving, determining the control strategy of the vehicle as: controlling the vehicle to perform a gentle deceleration operation;

[0117] In some embodiments, when the driving mode of the vehicle is automatic driving, a warning message can also be sent to the driver of the vehicle through the vehicle display module and / or voice module to enable the driver to understand the current automatic driving control strategy.

[0118] It should be noted that when the risk level of water splashing is medium, if the vehicle maintains its current driving state, water will splash onto moving objects in the future. Therefore, the vehicle should be controlled to take measures to decelerate smoothly to avoid splashing onto moving objects in the future.

[0119] Step S1056: When the risk level of water splashing risk is low risk and the driving mode is automatic driving, the control strategy of the vehicle is determined as: controlling the vehicle to maintain the current speed.

[0120] It should be noted that when the risk level of water splashing risk is low, if the vehicle maintains the current driving state, the probability of splashing water onto a moving object in the future is low, so the vehicle can be controlled to maintain the current speed.

[0121] The above-mentioned technical means illustrate the control strategies of vehicles under different risk levels. In this way, based on the risk level of splashing risk of mobile objects corresponding to the water accumulation area on the actual driving road of the vehicle, the control strategy corresponding to the risk level can be determined to effectively protect the mobile objects to safely pass through the splashing area.

[0122] In some embodiments, the above step S101, determining the location information of the water accumulation area on the road where the vehicle is currently traveling, may include steps S1011 to S1013:

[0123] Step S1011: determining first feature information in each image data in a bird's-eye view of the road based on the image data collected by multiple cameras;

[0124] Here, multiple cameras refer to at least one camera, such as 2, 3, 6 and 7. Bird's-Eye-View (BEV) refers to a method of projecting three-dimensional environment information onto a two-dimensional plane to display objects and terrain in the environment from a bird's-eye view.

[0125] In some embodiments, the method for determining the first feature information may include image feature extraction, depth estimation, and perspective conversion.

[0126] In some embodiments, the implementation of image feature extraction can be: after obtaining image data collected by multiple cameras, the vehicle's control device uses a deep learning algorithm (such as a convolutional neural network CNN) to perform feature extraction on the input multiple image data, so as to extract useful feature information from the multiple image data, such as edges, textures, colors, etc. These features will be used for subsequent depth estimation and perspective conversion.

[0127] In some embodiments, the implementation of depth estimation may be: first, a deep network is used to predict the depth information of each pixel in the plurality of image data, wherein the depth information is essential for converting the image features from the perspective projection perspective to the bird's-eye view perspective. Then, according to the prediction results of the deep network, a corresponding plurality of depth maps are generated, wherein the depth map is a matrix of the same size as the input plurality of image data, wherein the value of each element represents the depth information of the corresponding pixel.

[0128] In some embodiments, the perspective conversion may be implemented as follows: first, using the feature information of the depth map and the image data, the two-dimensional image features are promoted to the three-dimensional space, wherein this step generally involves combining the features of each pixel with its corresponding depth information to generate a three-dimensional feature point or voxel. Then, the three-dimensional feature point or voxel is projected onto the BEV plane, wherein this step involves converting the point or voxel in the three-dimensional space to the two-dimensional BEV plane while retaining its spatial position information. Finally, the projected features are aggregated on the BEV plane, wherein, since a ground position may correspond to feature points in multiple camera perspectives, these feature points need to be fused or averaged to generate the final first feature information.

[0129] Step S1012: determining second characteristic information in the third radar data in a bird's-eye view based on third radar data collected by a third radar installed on the vehicle; the first radar, the second radar, and the third radar are of different types;

[0130] Here, the third radar refers to a radar that transmits a detection signal to a target, and then compares the received signal reflected from the target with the transmitted signal, and after appropriate processing, can obtain relevant information about the target. For example, the third radar can be a laser radar.

[0131] In some embodiments, the method for determining the second feature information may include data preprocessing and feature extraction.

[0132] In some embodiments, data preprocessing may be implemented by: after acquiring the third radar data, the vehicle control device preprocesses the third radar data, including steps such as noise removal, filtering, and downsampling, so as to improve the quality and processing efficiency of the third radar data.

[0133] In some embodiments, the implementation of feature extraction can be: first, project the preprocessed third radar data onto the XY plane to generate a basic image of the radar BEV feature, wherein, in the process of data projection, it is necessary to select a suitable resolution, that is, to determine how large a range in the point cloud space corresponds to a pixel in the image. Then, on the projected BEV image, according to specific application requirements, select or design a suitable feature expression method, wherein common features include maximum height value, average height value, point cloud density, reflection intensity, etc., which can reflect information such as the shape, size and position of the target object. Finally, use a specific algorithm (such as a convolutional neural network, a deep learning model, etc.) to extract features from the BEV image to obtain second feature information.

[0134] Step S1013: Based on the first characteristic information and the second characteristic information, determine the location information of the water accumulation area on the road on which the vehicle is currently traveling.

[0135] In some embodiments, first, the first feature information and the second feature information are fused, wherein the fusion process can be point-level fusion or feature-level fusion; then, the fused feature information is subjected to three-dimensional target detection to identify target objects such as roads, vehicles, pedestrians, and waterlogged areas; finally, based on the waterlogged areas in the identified target objects, the location information of the waterlogged areas on the road on which the vehicle is currently traveling can be determined.

[0136] The above-mentioned technical means, firstly, determine the location information of the waterlogged area through the data collected by the third radar and at least one camera installed on the vehicle, which can make up for the shortcomings of a single device and improve the accuracy of calculating the location information of the waterlogged area.

[0137] In some embodiments, after the above step S1013, steps S1014 and S1015 may be included:

[0138] Step S1014: determining the location information of the waterlogged area determined based on the third radar data;

[0139] Step S1015: correcting the location information of the waterlogged area based on the radar data collected by the first radar, the radar data collected by the second radar, and the third radar data;

[0140] Among them, step S102, based on the location information of the water accumulation area, obtains the first radar data of the water accumulation area collected by the first radar installed on the vehicle, and the second radar data of the water accumulation area collected by the second radar, including: based on the corrected location information of the water accumulation area, obtains the first radar data of the water accumulation area collected by the first radar installed on the vehicle, and the second radar data of the water accumulation area collected by the second radar.

[0141] In some implementations, the location information and range information of the waterlogged area are preliminarily determined based on the height information and reflection intensity information of the radar data of the third radar.

[0142] In some embodiments, first, the first position information of the waterlogging area can be determined based on the radar data collected by the first radar, the second position information of the waterlogging area can be determined based on the radar data collected by the second radar, and the third position information of the waterlogging area can be determined based on the third radar data; then, the position information of the waterlogging area is corrected based on the first position information, the second position information, and the third position information.

[0143] In some embodiments, the implementation of determining the first position information of the waterlogged area based on the radar data collected by the first radar may be: first, performing data preprocessing on the radar data collected by the first radar; then, determining the distance value of each measuring point in the preprocessed radar data collected by the first radar; finally, by analyzing the changing trend of the distance value of each measuring point, the location of the waterlogged area can be preliminarily determined. For example, if the distance value of a certain area suddenly decreases, it may mean that there is waterlogging in the area.

[0144] In some embodiments, the implementation of determining the second position information of the waterlogged area based on the radar data collected by the second radar may be: first, performing data preprocessing on the radar data collected by the second radar; then, determining the distance value of each measuring point in the preprocessed radar data collected by the second radar; finally, by analyzing the changing trend of the distance value of each measuring point, the location of the waterlogged area can be preliminarily determined. For example, if the distance value of a certain area suddenly decreases, it may mean that there is waterlogging in the area.

[0145] In some embodiments, the implementation of determining the third position information of the water accumulation area based on the third radar data can be: first, preprocessing the third radar data; then, determining the height information and reflection intensity information of the preprocessed third radar data; finally, based on the height information and reflection intensity information of the third radar data, preliminarily judging the position and range of the water accumulation area.

[0146] The above-mentioned technical means, first of all, corrects the location information of the water accumulation area determined by the third radar through the data collected by the third radar, the first radar and the second radar installed on the vehicle, which can make up for the shortcomings of a single device and improve the accuracy of calculating the location information of the water accumulation area.

[0147] In some embodiments, the above step S101, determining the trajectory information of the moving object on the road the vehicle is currently traveling on, may include steps S1016 and S1017:

[0148] Step S1016: Determine information of a moving object on the road the vehicle is currently traveling on based on the first feature information and the second feature information;

[0149] In some embodiments, first, the first feature information and the second feature information are fused, wherein the fusion process can be point-level fusion or feature-level fusion; then, the fused feature information is subjected to three-dimensional target detection to identify target objects such as roads, vehicles, pedestrians, and waterlogged areas; finally, based on the mobile objects among the identified target objects, the information of the mobile objects on the road on which the vehicle is currently traveling can be determined, for example, the information of the mobile objects includes: information such as the position information of the mobile object, the moving speed of the mobile object, and the moving direction of the mobile object.

[0150] Step S1017: Based on the information of the moving object and the image data collected by multiple cameras, determine the trajectory information of the moving object on the road the vehicle is currently traveling on.

[0151] In some embodiments, after obtaining information about a moving object and image data collected by multiple cameras, the trajectory information of the moving object can be predicted through a Kalman filter (Kalman Filter) or an extended Kalman filter (EKF); wherein the trajectory information of the moving object refers to the set of moving positions of the moving object in the next few seconds.

[0152] In some implementations, after obtaining information about a moving object and image data captured by multiple cameras, the trajectory information of the moving object can be predicted through a deep learning trajectory prediction model.

[0153] In some embodiments, first, after detecting a moving object, it can be determined from a storage area of ​​a control device of the vehicle whether information of the moving trajectory is stored; then, after determining that information of the moving trajectory is stored in the storage area, historical information of the moving trajectory is obtained; finally, based on the historical information of the moving trajectory, current information of the moving object and image data captured by multiple cameras.

[0154] The above-mentioned technical means, firstly, determines the trajectory information of the moving object through the data collected by the third radar and at least one camera installed on the vehicle, which can make up for the shortcomings of a single device and improve the accuracy of calculating the trajectory information of the moving object.

[0155] The control method and system for the above-mentioned vehicle are explained below in conjunction with a specific embodiment. For ease of understanding, the ultrasonic radar is used as the first radar, the millimeter wave radar is used as the second radar, the laser radar is used as the third radar, and the VRU is used as the mobile object. An example is given below to introduce possible processes applicable to the embodiments of the present application. However, it is worth noting that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation on the present application.

[0156] In today's rapidly developing transportation field, autonomous driving is gradually becoming a mainstream trend. How to ensure the safe driving of autonomous vehicles in complex road environments has become an urgent problem to be solved.

[0157] Due to the unpredictable weather, especially the problem of road waterlogging caused by rainfall, it poses a hidden danger to driving safety that cannot be ignored. Waterlogging not only affects the vehicle's handling performance and increases the risk of skidding and loss of control, but may also hide potholes and obstacles on the road, further increasing the possibility of accidents. On the one hand, the depth and range of waterlogging on different sections of road vary. If a vehicle rashly drives into an area with too deep waterlogging, it may cause serious problems such as water ingress to the engine and electrical system failure, threatening the life safety of the driver and passengers in the car. On the other hand, the splashing water generated when the vehicle passes through the flooded road may affect VRUs, such as pedestrians and cyclists.

[0158] Existing detection technologies usually rely on a single type of sensor, such as cameras, radars, or ultrasonic sensors. However, these single sensors have obvious limitations. For example, the performance of cameras degrades in bad weather and poor lighting conditions; radars may be affected by electromagnetic interference and have limited detection accuracy for close-range objects; and ultrasonic sensors have a narrow detection range. In addition, most existing systems only focus on the safety of the vehicle itself, and lack comprehensive consideration of VRU protection and road surface water. They lack effective multi-sensor fusion and interactive visualization methods, and are unable to provide drivers or autonomous driving systems with comprehensive and intuitive road condition information, making it difficult to make accurate decisions and timely responses.

[0159] Therefore, there is an urgent need for a multi-sensor fusion vehicle control system that can accurately detect VRUs and road waterlogging at the same time. On the one hand, if the waterlogged area is too deep, an emergency alarm or brake can be issued; on the other hand, if the waterlogged area is shallow and can be safely passed, the surrounding VRUs can be detected, and the distance between the VRU and the vehicle, the water depth, and the current vehicle speed can be used to determine whether it is necessary to slow down and pass through, so as to protect the VRU from safely passing through the waterlogged area, thereby effectively improving the level of road traffic safety and reducing the occurrence of accidents. In addition, through interactive visualization, the current road conditions, information about the waterlogged area, and information about the VRU can be presented to the user in a clear and intuitive manner to assist the driver in controlling the vehicle's driving status.

[0160] First, a vehicle control system based on multi-sensor fusion provided by this embodiment is introduced. The system can efficiently and accurately detect VRU and road surface water, and present the detection results to the driver or the automatic driving system in real time through an interactive visual interface. Through this system, the problems raised in the above background technology can be effectively solved. Figure 2 As shown, the system mainly includes the following modules:

[0161] 1. Multi-sensor module 1:

[0162] Here, the vehicle control system installs six cameras 11, laser radar 12, millimeter wave radar 13 and ultrasonic radar 14 around the vehicle, among which the six cameras 11 are used to collect visual images of the surrounding environment and provide high-resolution visual information; the laser radar 12 is used to detect the position of objects on the road through the collected three-dimensional point cloud data; the millimeter wave radar 13 and ultrasonic radar 14 are used for long-distance and short-distance depth measurement respectively, and are particularly suitable for detecting waterlogged areas. The following is a detailed description of each sensor device of this module:

[0163] LiDAR 12: It is used to obtain high-resolution three-dimensional point cloud data, identify uneven areas on the road surface, such as the contours of waterlogged areas, and can accurately detect the shape and position of the VRU.

[0164] Millimeter-wave radar 13: Installing a millimeter-wave radar sensor on the vehicle enables it to emit millimeter-wave signals forward and receive reflected millimeter-wave signals. The millimeter-wave signals can penetrate the water surface to estimate the depth of puddles, thereby measuring the depth of water accumulation at medium and long distances.

[0165] Ultrasonic radar 14: An ultrasonic radar sensor is installed on the vehicle so that it can emit ultrasonic signals forward and receive reflected ultrasonic signals, so as to measure the depth of water at a close distance.

[0166] Camera 11: Six cameras are installed around the vehicle to cover a 360-degree view around the vehicle, which are used to obtain high-definition visual images and combine with radar data to further improve the accuracy of detection.

[0167] 2. Data processing and fusion module 2:

[0168] Here, the data processing and fusion module 2 includes a central processing unit (CPU) and a graphics processing unit (GPU), which are used to receive and process data from various sensors and generate an environmental perception model through data fusion technology. The functional modules of the data processing and fusion module 2 include a VRU and waterlogged area position detection module 21, a waterlogged area precise position and depth recognition module 23, and a VRU motion trajectory prediction module 22, which are used to implement the functions of the following units:

[0169] Data preprocessing unit: performs preliminary processing on the data collected by each sensor in the multi-sensor module, including denoising, correction, synchronization and other operations. Among them, the denoising operation can remove the noise in the data collected by each sensor, that is, those irrelevant or random fluctuations, which may interfere with the real information of the data; the correction operation can adjust the data collected by each sensor device to eliminate system errors or deviations, so that the data is closer to the real value; the synchronization operation can make the data collected by each sensor consistent in time, which is convenient for subsequent analysis.

[0170] Data fusion unit: Uses Kalman filtering, Bayesian reasoning and other algorithms to fuse data from different sensor devices to generate a more accurate environmental model.

[0171] Deep learning algorithm unit: The deep learning model is applied to the fused data to identify and classify VRUs and water accumulation, and the detection results can be updated in real time.

[0172] The functional modules of the data processing and fusion module 2 also include a danger zone determination module 24 for determining whether the VRU is in or about to enter a water splashing danger zone based on the depth information of the water accumulation zone, the predicted VRU trajectory information and the vehicle kinematic model.

[0173] 3. Interactive visualization and early warning module 3:

[0174] Display control unit: The processed detection results are displayed in real time through the vehicle display or HUD, including VRU location information, VRU trajectory information, depth information and area information of the water accumulation area, etc.

[0175] Warning system: When potential danger is detected, the vehicle's control system will trigger warning signals, including sound alarms, visual reminders, etc., and can automatically prompt the driver to take evasive measures.

[0176] Next, a vehicle control method based on multi-sensor fusion provided by this embodiment is introduced. This method ensures accurate detection of VRU and water accumulation through the collaborative work of multiple sensors, and improves driving safety through dynamic weight adjustment and automatic warning. The specific implementation method of this embodiment mainly includes: multi-sensor data acquisition, data fusion processing, VRU and water accumulation detection and classification, interactive visualization and warning, and system optimization.

[0177] 1. Multi-sensor data acquisition

[0178] LiDAR: It is used to collect 3D point cloud data. On the one hand, it performs multi-modal input target detection with the 6-camera images, mainly detecting the position and area of ​​the VRU and the water accumulation area on the BEV plane. On the other hand, it performs multi-modal input positioning and depth measurement of the water accumulation area with the millimeter-wave radar and ultrasonic radar. By combining these three sensors, they can make up for the shortcomings of a single sensor and improve the accuracy of puddle and depth measurement.

[0179] Millimeter-wave radar: used to estimate water depth by detecting the time difference between a signal reflected from the water surface and a signal that partially penetrates the water surface. This estimation relies on the millimeter-wave radar's ability to capture underwater reflected signals and is usually suitable for shallow puddles.

[0180] Ultrasonic radar: used for close-range detection, detecting the distance between the ultrasonic radar and the water surface, and supplementing the deficiencies of other sensors at close range.

[0181] Camera: Provides all-round visual images and works with LiDAR to detect VRUs and waterlogged areas.

[0182] 2. Data fusion processing

[0183] Data preprocessing: De-noising, synchronization and spatial alignment of multi-sensor data to ensure the accuracy and consistency of multi-sensor data.

[0184] Data fusion: Combining the characteristics of each sensor, through adaptive weight allocation, dynamically adjust the weight of each sensor according to the distance, and generate a comprehensive environmental perception model.

[0185] 3. Detection and classification of VRU and water accumulation

[0186] VRU detection: It is realized by using the 2D Occupancy Grid Algorithm (2DOCC) algorithm through vision + LiDAR to accurately identify the VRU and water accumulation area and obtain their position on the BEV plane.

[0187] Here, if Figure 2 As shown in the figure, the VRU and water accumulation area position detection module in the vehicle control system obtains the rough position information of the VRU and the water accumulation area based on the 6-camera image data and the laser radar data. The VRU and water accumulation area position detection module uses the images from the six cameras and the three-dimensional point cloud data of the laser radar for multi-modal input target detection, so that the position information and range information of the VRU and the water accumulation area in the BEV plane can be roughly determined.

[0188] During implementation, the 6-camera image data and lidar data are combined as multimodal inputs, and the multimodal 2DOCC algorithm (the occupancy task mainly focuses on analyzing the space occupancy in the environment, that is, which areas are occupied and which areas are not occupied, so as to assist the autonomous driving system in making decisions) is used to identify and locate VRUs (for example, pedestrians, cyclists, motorcyclists, etc.), waterlogged areas, and drivable areas. A feasible multimodal 2DOCC algorithm can be the Bevfusion algorithm, and the final 3D target detection Head is replaced with a 2DOCC Head. The specific model architecture is as follows: Figure 3 As shown, the final output results are the categories of VRU, water accumulation area and drivable area, as well as the positions of VRU, water accumulation area and drivable area in the BEV space respectively.

[0189] like Figure 3As shown, the application method of the model architecture includes the following steps: Step S301: Image data encoding; Here, the image data of the 6-camera image data is encoded, the feature information in the 6-camera image is extracted, and fusion is performed to obtain multi-view image features. Step S302: Perspective change; Here, the two-dimensional multi-view image features are mapped to the three-dimensional space to obtain the image BEV features. Step S303: Radar data encoding; Here, the lidar data is encoded, the corresponding point cloud image is obtained based on the lidar data, the feature information in the point cloud image is extracted, and fusion is performed to obtain the radar BEV features. Step S304: Feature fusion; Here, the radar BEV features and the image BEV features are input into the feature fusion module for feature fusion to obtain the fused BEV features. Step S305: 2DOCC Head; Here, the fused BEV features are input into the 2DOCC Head module, and the categories of VRU, waterlogged area and drivable area, as well as the positions of VRU, waterlogged area and drivable area in the BEV space are output respectively. Among them, the 2DOCC Head module is used to replace the traditional 3D object detection Head with the 2DOCC Head to meet the output requirements of the occupancy grid map, and generate an occupancy grid map based on the fused features to identify the categories of VRU, water accumulation area and drivable area, as well as the positions of VRU, water accumulation area and drivable area in the BEV space respectively.

[0190] Water accumulation detection: Use LiDAR, millimeter-wave radar, and ultrasonic radar to jointly detect puddles and their depths, and adaptively allocate sensor weights to achieve accurate measurement.

[0191] Here, if Figure 2 As shown, the precise position and depth identification module of the water accumulation area in the vehicle control system obtains the precise position information and depth information of the water accumulation area based on the rough position and size information of the water accumulation area.

[0192] During implementation, first, the rough location and size information of the waterlogged area sent by the VRU and the waterlogged area location detection module is received, and the depth information and precise location information of the waterlogged area are determined using the rough location and size information of the waterlogged area as prior information. Then, the depth information and precise location information of the waterlogged area are determined. The implementation of this process includes the following steps:

[0193] a) The auxiliary role of LiDAR

[0194] LiDAR is mainly used to identify the contours and precise location information of the waterlogged area, but not directly for depth measurement. The contours and precise location information of the waterlogged area are then passed to the millimeter wave radar and ultrasonic radar for further depth measurement.

[0195] b) Depth estimation of millimeter wave radar and ultrasonic radar

[0196] Millimeter-wave radar measurement: Millimeter-wave radar is used at medium and long distances. It has strong penetration ability and can estimate the depth of a puddle by detecting the time difference of the reflected signal. The depth information of the water area measured by the millimeter-wave radar (first depth information) is shown in formula (2).

[0197] Ultrasonic radar measurement: Ultrasonic radar is suitable for accurate measurement at close range. Its principle is similar to that of millimeter wave radar, but it has higher resolution at close range. The depth information of the water accumulation area measured by ultrasonic wave (second depth information) is shown in formula (1).

[0198] c) Multi-sensor result fusion

[0199] The depth measurement results of the millimeter-wave radar and the ultrasonic radar are fused to determine the depth information of the water accumulation area, see formula (3).

[0200] Estimated VRU trajectory information: For each detected VRU, the system uses a Kalman filter or an extended Kalman filter to predict the motion trajectory. By analyzing the VRU's historical position and velocity information, the control system can predict the VRU's future position. This can determine whether the VRU will enter the vehicle's driving path or splash hazard area based on the predicted VRU's trajectory information.

[0201] Here, if Figure 2 As shown, the trajectory information of the VRU is determined by the VRU motion trajectory prediction module in the vehicle control system.

[0202] During implementation, the trajectory information of the VRU can be predicted based on the current speed and direction of the VRU, the environmental context, and the historical motion data of the VRU through a Kalman filter or an extended Kalman filter or a trajectory prediction model based on deep learning. The prediction result of the trajectory information of the VRU is a possible position trajectory of the VRU in the next few seconds.

[0203] Among them, current speed and direction: the current motion state of the VRU is detected by the camera and lidar. Historical motion data: if the vehicle control system can obtain the historical motion data of the VRU and use it as input, the prediction accuracy of the VRU's trajectory information can be improved. Environmental context: combined with the surrounding environment, predict the possible behavior of the VRU, such as avoidance or continuing to go straight.

[0204] It should be noted that the trajectory prediction model also combines the vehicle's kinematic model, taking into account the vehicle's current speed, direction, and acceleration, further improving the accuracy of trajectory prediction.

[0205] Dangerous area judgment: Based on the size and depth of the water accumulation area, VRU motion estimation and vehicle kinematic model, it is judged whether the VRU is in or about to enter the dangerous area of ​​splashing water.

[0206] Here, if Figure 2 As shown, the dangerous area judgment module in the vehicle control system judges the dangerous area based on the VRU information, the water accumulation area information and the VRU trajectory information.

[0207] During implementation, first, the VRU information and waterlogging area information sent by the waterlogging area precise location and depth identification module, as well as the VRU trajectory information predicted by the VRU motion trajectory prediction module are received. The VRU information may include the running speed of the VRU, the running direction of the VRU, and other information; the waterlogging area information may include the depth information of the waterlogging area, the location information of the waterlogging area, and other information.

[0208] Then, by comparing the depth information of the flooded area with a preset safety threshold (the safety threshold D is usually determined based on the vehicle's wading performance and the maximum safe wading depth recommended by the manufacturer. For example, for ordinary passenger cars, the safety threshold may be set to 40 cm), it is determined whether there is a safety problem when the vehicle passes through the flooded area; if the depth information of the flooded area is greater than the safety threshold, it is determined that there is a safety problem when the vehicle passes through the flooded area, and the vehicle control system will immediately take emergency response measures such as emergency alarm prompts or automatic braking; if the depth information of the flooded area is less than the safety threshold, it is determined that there is no safety problem when the vehicle passes through the flooded area.

[0209] Next, if it is determined that there is no safety problem when the vehicle passes through the waterlogged area, it is determined whether the VRU is in or about to be in the splashing area based on the predicted VRU trajectory information, the depth information of the waterlogged area, and the vehicle's own kinematic model, and the splashing risk of the VRU is evaluated. If it is determined that the VRU is at risk of splashing, the driver is prompted through the interactive visualization and warning module and may take automatic braking or deceleration operations to avoid danger.

[0210] The splash area of ​​the water accumulation area refers to the area where water splashes may occur due to tire squeezing when a vehicle passes through the water accumulation at high speed. The size and shape of the splash area mainly depends on the following factors:

[0211] Vehicle speed (v): The faster the vehicle is, the wider the splash will be.

[0212] Water depth (D fused ): The deeper the water, the greater the height and range of the splashes.

[0213] Coefficient of friction between tire and road (k): The coefficient of friction affects the height and direction of water splashing.

[0214] Here, the splash area is modeled as a fan-shaped area extending in the direction of vehicle travel. The shape and size of the splash area can be found in formula (5).

[0215] After obtaining the splash area and VRU trajectory information, the danger area judgment module will evaluate the VRU splash risk. The evaluation process includes the following steps:

[0216] 1). Time to Collision (TTC) calculation

[0217] Here, TTC is a key parameter for evaluating the possibility of VRU entering the splash area. By predicting the trajectory information of VRU, the danger zone judgment module calculates the time t when VRU enters the splash area. VRU (i.e. the first time) and the time t when the vehicle arrives in the same area vehicle (i.e. second time):

[0218] t VRU : Estimate the time it takes for the VRU to reach the border of the splash zone based on the speed and direction of the VRU.

[0219] t vehicle :Based on the vehicle's current speed and route, calculate the time it takes for the vehicle to arrive at the corresponding location.

[0220] 2) Evaluate the risk of VRU splashing

[0221] Here, the hazardous area judgment module evaluates the VRU splash risk according to the following criteria:

[0222] High risk: If t VRU ≈t vehicle , that is, the VRU and the vehicle enter the splash area almost at the same time, and the system determines it as a high-risk situation.

[0223] Medium risk: If t VRU <t vehicle , but if the two are close, the system will determine it as a medium risk situation.

[0224] Low risk: If t VRU and t vehicle If the gap is large or the VRU has obvious avoidance behavior, the system determines it as low risk.

[0225] 4. Interactive visualization and early warning

[0226] Visual interface: Real-time display of VRU location, puddle depth and danger zone information, and intuitive presentation on the vehicle display or HUD.

[0227] Safety warning: On the one hand, when the VRU is detected in a dangerous area, the vehicle control system will trigger a warning to prompt the driver or automatically brake and slow down to avoid splashing accidents. On the other hand, if the water accumulation area is too deep, the vehicle control system will issue an emergency alarm or directly brake to avoid serious problems such as water ingress to the engine and electrical system failure, which threaten the life safety of the driver and passengers in the car.

[0228] Here, if Figure 2 As shown, the interactive visualization and warning module in the vehicle control system determines the vehicle control measures based on the assessment results of the VRU splash risk.

[0229] During implementation, first, the assessment result of the VRU splash risk sent by the dangerous area judgment module is received.

[0230] Then, the vehicle control measures are determined based on the VRU splash risk assessment results and the preset vehicle control strategy, where the vehicle control strategy includes driver prompts and control strategies under automatic driving control.

[0231] The control strategies for driver prompts include: High risk: The system issues an emergency prompt to the driver through the in-car display or audio warning, suggesting slowing down or braking. Medium risk: The system prompts the driver to pay attention to the situation ahead and suggests appropriate deceleration. Low risk: The system can choose not to issue a prompt, or only mark the potential splash area on the screen.

[0232] The control strategies under autonomous driving control include: High risk: The vehicle control system automatically performs emergency braking or deceleration operations to ensure that the vehicle does not splash water onto the VRU. Medium risk: The vehicle control system decelerates smoothly to reduce the possibility of splashing water. Low risk: The vehicle control system maintains the current speed and route and continues to monitor. Among them, in the autonomous driving mode, the vehicle control system can directly control the vehicle to operate according to the above control strategies.

[0233] Finally, the system displays all detected information and risk assessment results to the driver through the in-car display. The display content includes:

[0234] VRU and splash zone location: Graphically displays the location of the VRU and splash zone to help the driver understand the current risk.

[0235] Risk Level: Clearly identify the current risk level.

[0236] Recommended operation: Prompt the driver with appropriate operation suggestions, such as slowing down, avoiding, etc.

[0237] 5. System Optimization

[0238] Modular design: Modular design that supports independent upgrades and flexible configuration of different sensors and algorithms.

[0239] Adaptive Adjustment: The vehicle control system can adjust the detection strategy according to environmental conditions to ensure real-time and accuracy in different scenarios.

[0240] Based on the foregoing embodiments, this embodiment provides a vehicle control device, which includes the modules included and the units included in the modules, and can be implemented by a processor in the vehicle; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0241] Based on the above-mentioned embodiments, this embodiment provides a vehicle control device, wherein a first radar and a second radar are arranged on the vehicle; Figure 4 As shown, the control device 400 includes: a first acquisition module 410, which is used to acquire radar data collected by a first radar installed on the vehicle and radar data collected by a second radar installed on the vehicle; a first determination module 420, which is used to determine the trajectory information of the moving object on the road where the vehicle is currently traveling and the position information of the water accumulation area; a second acquisition module 430, which is used to acquire the first radar data of the water accumulation area collected by the first radar based on the position information of the water accumulation area and the radar data collected by the first radar, and to acquire the second radar data of the water accumulation area collected by the second radar based on the position information of the water accumulation area and the radar data collected by the second radar; the first radar and the second radar are of different types; a second determination module 440, which is used to determine the depth information of the water accumulation area based on the first radar data of the water accumulation area and the second radar data of the water accumulation area; a third determination module 450, which is used to determine the risk level of the risk of splashing water when the moving object passes through the road based on the driving speed of the vehicle and the trajectory information of the moving object, when it is determined that the vehicle can safely pass through the water accumulation area based on the depth information of the water accumulation area; a fourth determination module 460, which is used to determine the control strategy of the vehicle based on the risk level of splashing water risk.

[0242] In this embodiment, the vehicle control device includes: a fifth determination module, which is used to determine the vehicle control strategy as follows: outputting warning information through the vehicle's display module and / or controlling the vehicle to perform braking operations when it is determined that the vehicle cannot safely pass through the flooded area based on the depth information of the flooded area.

[0243] In this embodiment, the second determination module includes: a first determination unit, used to determine the first depth information of the waterlogged area based on the first radar data; a second determination unit, used to determine the second depth information of the waterlogged area based on the second radar data; and a third determination unit, used to determine the depth information of the waterlogged area based on the first depth information and the second depth information.

[0244] In this embodiment, the third determination module includes: a fourth determination unit, used to determine the splashing area triggered by the vehicle when passing through the water accumulation area based on the depth of the water accumulation area and the driving speed of the vehicle; a fifth determination unit, used to determine the first time when the moving object enters the splashing area based on the trajectory information of the moving object and the splashing area; a sixth determination unit, used to determine the second time when the vehicle enters the splashing area based on the driving speed of the vehicle and the splashing area; and a seventh determination unit, used to determine the risk level of the risk of splashing when the moving object passes the road based on the first time and the second time.

[0245] In this embodiment, the seventh determination unit includes: a first determination subunit, used to determine that the risk level of the risk of water splashing when the moving object passes through the road is high risk based on the first time, the second time and the first time threshold; a second determination subunit, used to determine that the risk level of the risk of water splashing when the moving object passes through the road is medium risk based on the first time, the second time and the second time threshold; the second time threshold is greater than the first time threshold; a third determination subunit, used to determine that the risk level of the risk of water splashing when the moving object passes through the road is low risk based on the first time, the second time and the third time threshold; the third time threshold is greater than or equal to the second time threshold.

[0246] In this embodiment, the fourth determination module includes: an eighth determination unit, which is used to determine, when the risk level of the water splashing risk is high risk, the control strategy of the vehicle as follows: issuing a warning message to the driver of the vehicle through the display module and / or voice module of the vehicle to remind the driver to slow down or brake; a ninth determination unit, which is used to determine, when the risk level of the water splashing risk is medium risk, the control strategy of the vehicle as follows: issuing a warning message to the driver of the vehicle through the display module and / or voice module of the vehicle to remind the driver to slow down; a tenth determination unit, which is used to determine, when the risk level of the water splashing risk is low risk, the control strategy of the vehicle as follows: controlling the vehicle not to issue a warning message, so that the vehicle drives according to the current driving state.

[0247] In this embodiment, the driving mode includes automatic driving, and the fourth determination module includes: an eleventh determination unit, which is used to determine the control strategy of the vehicle as: controlling the vehicle to perform emergency braking or emergency deceleration when the risk level of the water splashing risk is high risk and the driving mode is automatic driving; a twelfth determination unit, which is used to determine the control strategy of the vehicle as: controlling the vehicle to perform emergency braking or emergency deceleration when the risk level of the water splashing risk is medium risk and the driving mode is automatic driving; a thirteenth determination unit, which is used to determine the control strategy of the vehicle as: controlling the vehicle to maintain the current speed when the risk level of the water splashing risk is low risk and the driving mode is automatic driving.

[0248] In this embodiment, the first determination module includes: a fourteenth determination unit, which is used to determine the first characteristic information in each image data under the bird's-eye view of the road based on the image data collected by multiple cameras; a fifteenth determination unit, which is used to determine the second characteristic information in the third radar data under the bird's-eye view based on the third radar data collected by the third radar installed on the vehicle; the types of the first radar, the second radar and the third radar are different; a sixteenth determination unit, which is used to determine the location information of the flooded area on the road where the vehicle is currently traveling based on the first characteristic information and the second characteristic information.

[0249] In this embodiment, the first determination module includes: a seventeenth determination unit, used to determine the location information of the water accumulation area determined based on the third radar data; an eighteenth determination unit, used to correct the location information of the water accumulation area based on the radar data collected by the first radar, the radar data collected by the second radar and the third radar data; wherein, the second acquisition module includes: based on the corrected location information of the water accumulation area, acquiring the first radar data of the water accumulation area collected by the first radar installed on the vehicle, and the second radar data of the water accumulation area collected by the second radar.

[0250] This embodiment also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a vehicle control device, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0251] This embodiment also proposes a computer program, including a computer program or instructions, which implements part or all of the steps in the above method when executed by the control device of the vehicle.

[0252] This embodiment also proposes a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0253] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of the present invention for understanding.

Claims

1. A vehicle control method, characterized in that: The method comprises: Determine the trajectory information of the moving objects on the road the vehicle is currently traveling on and the location information of the waterlogged area; Acquire first radar data of the water accumulation area collected by a first radar installed on a vehicle and second radar data of the water accumulation area collected by a second radar based on the position information of the water accumulation area; the first radar and the second radar are of different types; Determining depth information of the waterlogging area based on the first radar data of the waterlogging area and the second radar data of the waterlogging area; In a case where it is determined based on the depth information of the waterlogged area that the vehicle can safely pass through the waterlogged area, determining the risk level of water splashing risk of the moving object when passing through the road based on the driving speed of the vehicle and the trajectory information of the moving object; Based on the risk level of the water splash risk, a control strategy of the vehicle is determined.

2. The method according to claim 1, characterized in that: The method comprises: When it is determined that the vehicle cannot safely pass through the flooded area based on the depth information of the flooded area, the control strategy of the vehicle is determined as: outputting warning information through the display module and / or voice module of the vehicle and / or controlling the vehicle to perform braking operations.

3. The method according to claim 1, characterized in that: The determining the depth information of the waterlogged area based on the first radar data of the waterlogged area and the second radar data of the waterlogged area includes: Determining first depth information of the waterlogged area based on the first radar data; Determining second depth information of the waterlogged area based on the second radar data; Determine depth information of the water accumulation area based on the first depth information and the second depth information.

4. The method according to claim 1, characterized in that: Determining, based on the driving speed of the vehicle and the trajectory information of the moving object, a risk level of water splashing risk when the moving object passes through the road, includes: Determining a splashing area triggered when the vehicle passes through the water accumulation area based on the depth of the water accumulation area and the driving speed of the vehicle; Determining a first time when the moving object enters the splash area based on the trajectory information of the moving object and the splash area; Determining a second time when the vehicle enters the splash zone based on the vehicle's travel speed and the splash zone; Based on the first time and the second time, a risk level of water splashing risk when the moving object passes through the road is determined.

5. The method according to claim 4, characterized in that: The determining, based on the first time and the second time, a risk level of water splashing when the moving object passes through the road includes: Based on the first time, the second time, and the first time threshold, determining that the risk level of water splashing risk when the mobile object passes through the road is high risk; Based on the first time, the second time and a second time threshold, determining that the risk level of water splashing risk when the moving object passes through the road is medium risk; the second time threshold is greater than the first time threshold; Based on the first time, the second time and a third time threshold, it is determined that the risk level of water splashing risk when the moving object passes through the road is low risk; the third time threshold is greater than or equal to the second time threshold.

6. The method according to any one of claims 1 to 5, characterized in that: The determining of the control strategy of the vehicle based on the risk level of the water splash risk includes: In the case where the risk level of the splash risk is high, determining the control strategy of the vehicle is: sending a warning message to the driver of the vehicle through the display module and / or voice module of the vehicle to remind the driver to decelerate or brake; When the risk level of the splash risk is medium risk, determining the control strategy of the vehicle is: sending a warning message to the driver of the vehicle through the display module and / or voice module of the vehicle to remind the driver to perform a deceleration operation; When the risk level of the water splashing risk is low risk, the control strategy of the vehicle is determined as: controlling the vehicle not to issue a warning message so that the vehicle travels according to the current driving state.

7. The method according to any one of claims 1 to 5, characterized in that: The driving mode includes automatic driving, and the determining of the control strategy of the vehicle based on the risk level of the water splash risk includes: When the risk level of the water splash risk is high risk and the driving mode is automatic driving, determining the control strategy of the vehicle is: controlling the vehicle to perform an emergency braking or emergency deceleration operation; When the risk level of the water splash risk is medium risk and the driving mode is automatic driving, determining the control strategy of the vehicle is: controlling the vehicle to perform a gentle deceleration operation; When the risk level of the water splash risk is low risk and the driving mode is automatic driving, the control strategy of the vehicle is determined as: controlling the vehicle to maintain the current vehicle speed.

8. The method according to any one of claims 1 to 5, characterized in that: Determine the location information of the waterlogged area on the road the vehicle is currently traveling on, including: Determining first feature information in each of the image data in a bird's-eye view of the road based on the image data collected by the plurality of cameras; determining second characteristic information in the third radar data under the bird's-eye view based on third radar data collected by a third radar installed on the vehicle; the first radar, the second radar and the third radar are of different types; Based on the first feature information and the second feature information, position information of a water accumulation area on the road on which the vehicle is currently traveling is determined.

9. The method according to claim 8, characterized in that: The method further comprises: Determining location information of the waterlogged area determined based on the third radar data; Based on the radar data collected by the first radar, the radar data collected by the second radar, and the third radar data, correcting the position information of the waterlogged area; Wherein, obtaining first radar data of the water accumulation area collected by a first radar installed on a vehicle and second radar data of the water accumulation area collected by a second radar based on the position information of the water accumulation area includes: Based on the corrected position information of the water accumulation area, first radar data of the water accumulation area collected by a first radar installed on the vehicle and second radar data of the water accumulation area collected by a second radar are obtained.

10. A vehicle control device, characterized in that: The vehicle is provided with a first radar and a second radar; the control device comprises: A first acquisition module, used to acquire radar data collected by the first radar installed on the vehicle and radar data collected by the second radar installed on the vehicle; A first determination module is used to determine the trajectory information of the moving object and the position information of the water accumulation area on the road where the vehicle is currently traveling; a second acquisition module, configured to acquire first radar data of the waterlogged area collected by the first radar based on the location information of the waterlogged area and the radar data collected by the first radar, and to acquire second radar data of the waterlogged area collected by the second radar based on the location information of the waterlogged area and the radar data collected by the second radar; the first radar and the second radar are of different types; A second determination module is used to determine the depth information of the waterlogging area based on the first radar data of the waterlogging area and the second radar data of the waterlogging area; A third determination module is used to determine the risk level of water splashing risk of the moving object when passing through the road based on the driving speed of the vehicle and the trajectory information of the moving object, when it is determined that the vehicle can safely pass through the water accumulation area based on the depth information of the water accumulation area; The fourth determination module is used to determine the control strategy of the vehicle based on the risk level of the water splash risk.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a control device of a vehicle, the steps of the method described in any one of claims 1 to 9 are implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a control device of a vehicle, the steps of the method according to any one of claims 1 to 9 are implemented.

Images