Vehicle driving planning method and device, electronic equipment and storage medium

Abstract

The invention discloses a vehicle driving planning method and device, electronic equipment and a storage medium. The method comprises the steps that multi-mode sensing data and vehicle environment data are acquired; the multi-modal perception data comprises multiple perception data of the vehicle; determining a fusion weight based on the vehicle environment data and the plurality of perceptual data; fusing the multiple pieces of sensing data based on the fusion weight to obtain fused environment sensing data; determining a scene danger level based on the multiple pieces of perception data and the fused environment perception data; and performing automatic driving planning based on the scene danger level and the fusion environment perception data. In the embodiment of the invention, the problems of poor environmental adaptability and insufficient static fusion robustness of a single sensor are solved, and the accuracy of environmental perception in a complex scene is improved; the scene grading decision improves the response efficiency in an emergency scene, reduces the collision risk, and enhances the safety of automatic driving.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a vehicle driving planning method and device, an electronic device and a storage medium. BACKGROUND

[0002] The traditional vehicle braking system mainly relies on the subjective judgment and operation of the driver. However, in the case of sudden danger, the driver may not be able to take effective braking measures in time due to insufficient reaction time, operation errors or distraction, etc., leading to accidents or aggravating the consequences of accidents.

[0003] With the development of automatic driving technology, although some warning and braking systems based on a single sensor have appeared, such as obstacle detection using only visual sensors or millimeter wave radars, in actual road environments, a single sensor is greatly affected by environmental interference. In bad weather conditions such as rain, fog and strong light, the sensing accuracy and reliability decrease significantly, it is difficult to obtain comprehensive and accurate vehicle surrounding environment information, and problems such as missed detection and false detection are likely to occur, resulting in insufficient accuracy of warning and braking decisions, and generally lacking a rapid decision-making mechanism for sudden high-risk scenarios, leading to insufficient safety redundancy of the automatic driving system and a large safety risk. SUMMARY

[0004] In order to solve the existing technical problems, the present application provides a vehicle driving planning method, device, electronic device and storage medium, which combines vehicle environment data and the characteristics of each perception data to dynamically generate a multi-modal data fusion weight adapted to the current scene, fuses the multi-perception data according to the fusion weight, and outputs precise and unified fusion environment perception data, solving the problems of poor environmental adaptability of a single sensor and insufficient robustness of static fusion, improving the accuracy of environment perception in complex scenarios; based on comprehensive analysis of the original perception data and the fusion data, determining the scene danger level, formulating an automatic driving strategy adapted to the scene according to the danger level and the fusion perception data, the scene classification decision improves the response efficiency in sudden scenarios, reduces the collision risk, and strengthens the safety of automatic driving.

[0005] In a first aspect, the embodiments of the present application provide a vehicle driving planning method, which comprises: obtaining multi-modal perception data and vehicle environment data; the multi-modal perception data comprises a plurality of perception data of the vehicle; determining a fusion weight based on the vehicle environment data and the plurality of perception data; fusing the plurality of perception data based on the fusion weight to obtain fusion environment perception data; determining a scene danger level based on the plurality of perception data and the fusion environment perception data; performing automatic driving planning based on the scene danger level and the fusion environment perception data.

[0006] In an optional embodiment, the vehicle environment data comprises weather data, illumination data and traffic flow density data; determining the fusion weight based on the vehicle environment data and the plurality of perception data, comprising: performing feature extraction on the weather data, the illumination data and the traffic flow density data to obtain current environment features; obtaining a prior confidence probability of the plurality of perception data; determining a conditional probability of the plurality of perception data under the current environment based on the current environment features and a preset environment confidence mapping relationship; determining a posterior confidence probability based on the Bayes formula, the prior confidence probability and the conditional probability; determining the fusion weight based on the posterior confidence probability and the plurality of perception data.

[0007] In an optional embodiment, determining the fusion weight based on the posterior confidence probability and the plurality of perception data, comprising: performing feature extraction on the plurality of perception data to obtain multi-modal features; inputting the posterior confidence probability and the multi-modal features into a neural network model containing an attention mechanism to output the fusion weight.

[0008] In an optional embodiment, fusing the plurality of perception data based on the fusion weight to obtain fused environment perception data, comprising: performing time synchronization on the plurality of perception data to obtain a plurality of time-synchronized perception data; the plurality of perception data comprises image data, millimeter wave radar data, laser radar data and vehicle attitude position data; determining the fused environment perception data based on the fusion weight and the plurality of perception data.

[0009] In an optional embodiment, the plurality of perception data comprises vehicle motion state data; determining a scene danger level based on the plurality of perception data and the fused environment perception data, comprising: constructing a danger probability prediction model based on a transformer model and a time sequence feature extraction model; inputting the vehicle motion state data and the fused environment perception data into the danger probability prediction model to output a danger probability value of a preset time period; determining the scene danger level based on the danger probability value.

[0010] In an optional embodiment, determining the scene danger level based on the danger probability value, comprising: if the danger probability value is less than a first danger probability threshold, determining a low danger level as the scene danger level; or; If the risk probability value is greater than or equal to the first risk probability threshold value and less than or equal to the second risk probability threshold value, a medium risk level is determined as the scene risk level; the second risk probability threshold value is greater than the first risk probability threshold value; or If the risk probability value is greater than the second risk probability threshold value, a high risk level is determined as the scene risk level.

[0011] In an optional embodiment, based on the scene risk level and the fused environmental perception data, automatic driving planning is performed, including: If the scene risk level is a low risk level, a first driving planning strategy is executed at a first frequency; the first driving planning strategy is to generate a candidate path based on the fused environmental perception data and a path planning algorithm, and generate an automatic driving instruction based on the candidate path and traffic rule information; or If the scene risk level is a medium risk level, the first driving planning strategy is executed at a second frequency; the second frequency is greater than the first frequency; or If the scene risk level is a high risk level, a second driving planning strategy is executed; the first driving planning strategy is to determine a target evasive action from a pre-compiled evasive action set based on the fused environmental perception data, and generate an automatic driving instruction based on the target evasive action; the pre-compiled evasive action set includes a plurality of evasive actions; the plurality of evasive actions correspond one-to-one to the plurality of fused environmental perception data.

[0012] In a second aspect, the embodiments of the present application provide a vehicle driving planning device, the device comprising: An acquisition module is configured to acquire multi-modal perception data and vehicle environmental data; the multi-modal perception data includes a plurality of perception data of a vehicle; A first determination module is configured to determine a fusion weight based on the vehicle environmental data and the plurality of perception data; A data fusion module is configured to fuse the plurality of perception data based on the fusion weight to obtain fused environmental perception data; A second determination module is configured to determine a scene risk level based on the plurality of perception data and the fused environmental perception data; A driving planning module is configured to perform automatic driving planning based on the scene risk level and the fused environmental perception data.

[0013] In a third aspect, the embodiments of the present application provide an electronic device, which comprises a processor and a memory; the memory stores at least one instruction, at least one program, a code set or an instruction set; the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the vehicle driving planning method of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, and the computer-readable storage medium stores at least one instruction or at least one program. The at least one instruction or the at least one program is loaded and executed by a processor to implement the vehicle driving planning method in the first aspect.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product or a computer program, and the computer program product or the computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to enable the computer device to implement the vehicle driving planning method in the first aspect.

[0016] The vehicle driving planning method, the device, the electronic device, and the storage medium provided by the embodiments of the present application have the following technical effects: The vehicle environment data and the multi-modal perception data are obtained, the multi-modal perception data includes multiple perception data of the vehicle, the fusion weight is determined based on the vehicle environment data and the multiple perception data, the multiple perception data are fused based on the fusion weight to obtain the fused environment perception data, the scene danger level is determined based on the multiple perception data and the fused environment perception data, and the automatic driving planning is performed based on the scene danger level and the fused environment perception data. In the embodiments of the present application, the multi-modal data fusion weight adaptive to the current scene is dynamically generated by combining the vehicle environment data with the characteristics of the perception data, the multiple perception data are fused according to the fusion weight, and the accurate and unified fused environment perception data is output. The problem of poor environment adaptability of a single sensor and insufficient robustness of static fusion is solved, and the accuracy of environment perception in a complex scene is improved. The scene danger level is determined based on the comprehensive analysis of the original perception data and the fused data, the automatic driving strategy adaptive to the scene is formulated according to the danger level and the fused perception data, the response efficiency in a sudden scene is improved by scene grading decision, the collision risk is reduced, and the safety of automatic driving is strengthened. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, and the advantages thereof, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on these drawings.

[0018] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application; Figure 2 is a flowchart of a vehicle driving planning method provided by an embodiment of the present application Figure 1 ; Figure 3 is a flowchart of a vehicle driving planning method provided by an embodiment of the present application Figure 2 ; Figure 4 is a structural diagram of a vehicle driving planning device provided by an embodiment of the present application. Figure 5 is a hardware structural diagram of a server of a vehicle driving planning method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0020] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0021] Please refer to Figure 1 , Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application, including information collection equipment 101, vehicle-mounted server 102 and execution equipment 103.

[0022] In a possible embodiment, the information collection equipment 101 is configured to collect multi-modal perception data and vehicle environment data.

[0023] Specifically, the information collection device 101 can include a camera for collecting high-resolution visible light images of the environment, providing visual information such as target detection, lane line recognition, and traffic sign recognition; can also include a millimeter wave radar for detecting the distance, relative speed, and azimuth information of targets around the vehicle, and has all-weather working capability; can also include a laser radar for generating high-precision three-dimensional point cloud data for accurate mapping, obstacle contour extraction, and spatial ranging; and can also include a Global Positioning System (GPS) and an Inertial Measurement Unit (IMU) for providing global positioning information of the vehicle and vehicle attitude data such as acceleration, angular velocity, and heading angle, and realizing high-precision positioning.

[0024] Further, the information collection device 101 can also collect environmental information such as weather, illumination, and traffic congestion, and specifically includes a weather sensor for collecting weather information, including a rain amount sensor for identifying the amount of rain in a rainy or snowy day, a humidity sensor for detecting air humidity, and a barometric pressure sensor for detecting air pressure to assist in determining weather changes. The information collection device 101 also includes an illumination sensor for collecting illumination information, accurately detecting ambient light intensity, identifying strong light, backlight, night scenes, and outputting illumination data; and a traffic flow detection device for collecting traffic congestion, which can be combined with a camera to count vehicles through image recognition and millimeter wave radar to assist in counting vehicles, and output congestion level information based on algorithms.

[0025] In one possible embodiment, the information collection device 101 is in communication connection with the vehicle-mounted server 102, and the vehicle-mounted server 102 receives the multi-modal perception data and vehicle environment data sent by the information collection device 101.

[0026] In one possible embodiment, the vehicle-mounted server 102 obtains multi-modal perception data and vehicle environment data; the multi-modal perception data includes a plurality of perception data of the vehicle; a fusion weight is determined based on the vehicle environment data and the plurality of perception data; the plurality of perception data is fused based on the fusion weight to obtain fused environmental perception data; a scene danger level is determined based on the plurality of perception data and the fused environmental perception data; and automatic driving planning is performed based on the scene danger level and the fused environmental perception data.

[0027] In one possible embodiment, the execution device 103 receives the driving strategy planned by the vehicle-mounted server 102, executes driving actions such as obstacle avoidance, acceleration, and deceleration, and performs automatic driving or assisted driving based on multi-modal perception data.

[0028] In the embodiment of the present application, the multi-modal data fusion weight adaptive to the current scene is dynamically generated by combining the vehicle environment data with the characteristics of each perception data, the multi-perception data is fused according to the fusion weight, and the accurate and unified fusion environment perception data is output, thereby solving the problems of poor environment adaptability of a single sensor and insufficient robustness of static fusion, improving the accuracy of environment perception in a complex scene; based on comprehensive analysis of the original perception data and the fusion data, the scene danger level is determined, and the automatic driving strategy adaptive to the scene is formulated according to the danger level and the fusion perception data, thereby improving the response efficiency in a sudden scene, reducing the collision risk, and strengthening the safety of automatic driving The following introduces a specific embodiment of a vehicle driving planning method of the present application, Figure 2 is a flowchart of a vehicle driving planning method provided by an embodiment of the present application Figure 1 The present specification provides method operation steps such as embodiments or flowcharts, but more or fewer operation steps can be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one of the many execution orders of the steps, and does not represent the only execution order. In actual system or server product execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel (for example, in a parallel processor or multi-thread processing environment). Specifically, as shown in Figure 2 may include: S201: acquiring multi-modal perception data and vehicle environment data; the multi-modal perception data includes multiple perception data of the vehicle.

[0029] S202: determining a fusion weight based on the vehicle environment data and the multiple perception data.

[0030] S203: fusing the multiple perception data based on the fusion weight to obtain fusion environment perception data.

[0031] S204: determining a scene danger level based on the multiple perception data and the fusion environment perception data.

[0032] S205: performing automatic driving planning based on the scene danger level and the fusion environment perception data.

[0033] Figure 3 is a flowchart of a vehicle driving planning method provided by an embodiment of the present application Figure 2 The method can include: S301: acquiring multi-modal perception data and vehicle environment data.

[0034] In the embodiments of the present application, the multi-modal perception data includes multiple perception data of the vehicle. The perception data can be visual perception data such as environment visible light images, target (pedestrian or vehicle) images, lane line and traffic sign images collected by a camera, radar perception data such as surrounding target distance, relative speed, azimuth information collected by a millimeter wave radar, three-dimensional point cloud data collected by a laser radar, global positioning data, acceleration, angular velocity, heading angle attitude data of the vehicle collected by a GPS / IMU, and the like.

[0035] S302: determining a fusion weight based on the vehicle environment data and the multiple perception data.

[0036] In an optional embodiment, one of a weighted Bayesian fusion or a deep neural network fusion is used to adaptively adjust the weight distribution of each modal data in the fusion.

[0037] In an optional embodiment, the weighted Bayesian fusion and the deep neural network fusion can be combined to adaptively adjust the weight distribution of each modal data in the fusion, which is described in detail as follows.

[0038] In an optional embodiment, the vehicle environment data includes weather data, illumination data and traffic flow density data. The weather data and the illumination data are obtained by sensors, and the traffic flow density data is obtained by a traffic flow density analysis model.

[0039] In an optional embodiment, the fusion weight is determined based on the vehicle environment data and the multiple perception data, including: S3021: performing feature extraction on the weather data, the illumination data and the traffic flow density data to obtain current environment features.

[0040] A single perception mode is easily affected by the environment: in foggy, rainy, night and other scenes, the imaging quality of the camera is reduced; some material objects have poor reflection to the laser radar; and the millimeter wave radar may produce false targets in complex reflection scenes. Therefore, features need to be extracted from the original environment data to quantify the influence of the environment on the performance of the sensor. Specifically, the weather data, the illumination data and the traffic flow density data are processed by numericalization, normalization, encoding and the like, and are converted into an environment feature vector.

[0041] S3022: obtaining prior confidence probabilities of the multiple perception data.

[0042] Secondly, the reliability prior knowledge of the sensor corresponding to each perception data in a general case is obtained. The multiple perception data come from different sensors such as a camera, a laser radar and a millimeter wave radar. The prior confidence probability of each sensor is obtained based on historical performance statistics or experimental calibration, and represents the probability of providing accurate perception results by the sensor under the default condition.

[0043] S3023: Determine the conditional probability of the plurality of perception data in the current environment based on the current environment feature and the preset environment confidence mapping relationship.

[0044] In a possible embodiment, the preset environment confidence mapping relationship is pre-established through a large number of experiments or data analysis, reflecting the influence law of environmental factors on sensor performance. For example, in a heavy rain environment, the camera conditional probability can be reduced to 0.6, the laser radar is reduced to 0.7, and the millimeter wave radar remains 0.9.

[0045] The preset environment confidence mapping relationship can be a lookup table or a function model, different environment features correspond to different confidence, by combining the current environment feature and the preset environment confidence mapping relationship, the current environment feature can be mapped to the conditional probability of each sensor in the current environment.

[0046] S3024: Determine the posterior confidence probability based on the Bayes formula, the prior confidence probability and the conditional probability.

[0047] Combining prior knowledge and current environmental conditions, update the evaluation of the reliability of each sensor. Specifically, using the Bayes formula, the prior confidence probability and the conditional probability are fused to calculate the posterior confidence probability. For each sensor, the posterior probability is the product of the prior probability and the conditional probability.

[0048] S3025: Determine the fusion weight based on the posterior confidence probability and the plurality of perception data.

[0049] In an optional embodiment, determining the fusion weight based on the posterior confidence probability and the plurality of perception data includes: performing feature extraction on the plurality of perception data to obtain multi-modal features, inputting the posterior confidence probability and the multi-modal features into a neural network model containing an attention mechanism, and outputting the fusion weight.

[0050] Finally, according to the updated sensor reliability and the features of the plurality of perception data, a fusion weight for fusing the plurality of perception data is generated.

[0051] Specifically, the perception data of each sensor is feature extracted to obtain a multi-modal feature vector.

[0052] The posterior confidence probability and the multi-modal features are input into a neural network containing an attention mechanism, the importance of different sensor features is dynamically learned through the attention module, and a set of normalized fusion weights are output by integrating the posterior confidence.

[0053] Among them, the sensor with high weight occupies a larger proportion in the fusion, and the sensor with low weight occupies a smaller proportion in the fusion. Through the combination of the Bayesian update of environmental perception and the data-driven attention mechanism, both the environmental prior knowledge and the real-time data characteristics are considered, so that the fusion system can adapt to the complex and changeable driving environment and improve the perception accuracy and reliability.

[0054] The present application introduces environmental characteristics such as weather, illumination, and traffic density into the weight distribution of multi-modal perception data in real time, realizes dynamic weight adjustment by using weighted Bayesian fusion and / or deep neural networks with attention mechanism, and avoids the problem of performance degradation of traditional fixed weight fusion in a specific environment.

[0055] S303: Fuse the plurality of perception data based on the fusion weight to obtain the fused environmental perception data.

[0056] In an optional embodiment, the plurality of perception data is fused based on the fusion weight to obtain the fused environmental perception data, including: S3031: Time-synchronize the plurality of perception data to obtain a plurality of time-synchronized perception data.

[0057] In the embodiment of the present application, the plurality of perception data specifically includes image data, millimeter wave radar data, laser radar data, and vehicle attitude and position data.

[0058] Before fusing the plurality of perception data based on the fusion weight, the different perception data needs to be accurately aligned by using a time stamp. Specifically, a hardware time synchronization (Precision Time Protocol, PTP) or a software interpolation synchronization method is adopted to ensure the time sequence consistency of data fusion.

[0059] S3032: Determine the fused environmental perception data based on the fusion weight and the plurality of perception data.

[0060] The plurality of time-synchronized perception data is obtained according to the fusion weight to obtain the fused environmental perception data.

[0061] S304: Determine the scene danger level based on the plurality of perception data and the fused environmental perception data.

[0062] In an optional embodiment, the plurality of perception data includes vehicle motion state data. The vehicle motion state data is vehicle speed data, acceleration data, etc.

[0063] In an optional embodiment, the scene danger level is determined based on the plurality of perception data and the fused environmental perception data, including: S3041: Construct a danger probability prediction model based on a transformer model and a time sequence feature extraction model.

[0064] S3042: input the vehicle motion state data and the fused environment perception data into the danger probability prediction model, and output a danger probability value in a preset time period.

[0065] S3043: determine a scene danger level based on the danger probability value.

[0066] Firstly, a danger probability prediction model is constructed by using a Transformer structure combined with a time sequence feature extraction model (such as a Temporal Convolutional Network, TCN) to capture the trend of environmental state changes. The danger probability prediction model can predict the potential danger degree in a short time (1-3 seconds), and dynamically adjust the perception and decision response frequency of the system based on the prediction result. The vehicle motion state data and the fused environment perception data are input into the danger probability prediction model, and a danger probability value in a preset time period (1 second, 2 seconds, or 3 seconds in the future) is output.

[0067] In an optional embodiment, if the danger probability value is less than a first danger probability threshold, a low danger level is determined as the scene danger level.

[0068] In an optional embodiment, if the danger probability value is greater than or equal to the first danger probability threshold and less than or equal to a second danger probability threshold, a medium danger level is determined as the scene danger level.

[0069] The second danger probability threshold is greater than the first danger probability threshold.

[0070] In an optional embodiment, if the danger probability value is greater than the second danger probability threshold, a high danger level is determined as the scene danger level.

[0071] For example, when the danger probability value is less than 30%, a low danger level is determined as the scene danger level. When the danger probability value is between 30% and 70%, a medium danger level is determined as the scene danger level. When the danger probability value is greater than 70%, a high danger level is determined as the scene danger level.

[0072] S305: perform automatic driving planning based on the scene danger level and the fused environment perception data.

[0073] In an optional embodiment, if the scene danger level is a low danger level, a first driving planning strategy is executed at a first frequency.

[0074] In the embodiments of the present application, the first frequency is a lower frequency, and the system maintains a regular perception refresh rate and a decision frequency.

[0075] In an optional embodiment, if the scene danger level is a medium danger level, the first driving planning strategy is executed at a second frequency.

[0076] In the embodiments of the present application, the second frequency is greater than the first frequency, and the system moderately increases the refresh rate and the frequency after the danger level is increased.

[0077] In an optional embodiment, if the scene danger level is a high danger level, a second driving planning strategy is executed.

[0078] In the embodiments of the present application, when the current scene danger level is a high danger level, a fast decision strategy, that is, a second driving planning strategy, is executed.

[0079] The scene priority prediction result is fed back to the adaptive decision engine in real time to trigger mode switching.

[0080] In an optional embodiment, the first driving planning strategy is to generate a candidate path based on fused environmental perception data and a path planning algorithm, and to generate an automatic driving instruction based on the candidate path and traffic rule information. Specifically, an optimal driving path is generated based on a path planning algorithm (such as A*, Dijkstra, or RRT*), and a smooth driving decision is output in combination with traffic rules (speed limit, priority of passage, red light control), and the first driving planning strategy is applicable to low and medium danger levels (low and medium priority scenes).

[0081] In an optional embodiment, the second driving planning strategy is to determine a target evasive action from a pre-compiled evasive action set based on fused environmental perception data, and to generate an automatic driving instruction based on the target evasive action. The pre-compiled evasive action set includes a plurality of evasive actions, and the plurality of evasive actions correspond one-to-one to a plurality of fused environmental perception data.

[0082] In a high-risk scene, a pre-compiled evasive action set is directly called, including emergency braking, emergency lane changing, and rapid bypassing, and the evasive action set is pre-generated through a large number of simulations and real vehicle tests to ensure that the execution delay is less than 100 milliseconds.

[0083] By quickly calling the pre-set emergency evasive action set, part of the path planning calculation process can be bypassed to reduce the reaction time. The perception accuracy and decision timeliness of the autonomous vehicle can be significantly improved in a complex dynamic traffic environment, thereby effectively reducing the risk of traffic accidents.

[0084] Finally, the decision result is converted into a bottom-layer execution command (steering angle, acceleration, braking force, etc.) and sent to the execution mechanism of the vehicle.

[0085] The embodiments of the present application also provide a vehicle driving planning device, Figure 4 A structural schematic diagram of a vehicle driving planning device provided by the embodiments of the present application is shown in FIG. 4. Figure 4 As shown in FIG. 4, the device 400 includes: The acquisition module 401 is configured to acquire multi-modal perception data and vehicle environment data; the multi-modal perception data comprises a plurality of perception data of a vehicle; The first determination module 402 is configured to determine a fusion weight based on the vehicle environment data and the plurality of perception data. The data fusion module 403 is configured to fuse the plurality of perception data based on the fusion weight to obtain fused environment perception data. The second determination module 404 is configured to determine a scene danger level based on the plurality of perception data and the fused environment perception data. The driving planning module 405 is configured to perform automatic driving planning based on the scene danger level and the fused environment perception data.

[0086] In an optional embodiment, the vehicle environment data comprises weather data, illumination data and traffic flow density data; and further comprises: The first feature extraction module is configured to perform feature extraction on the weather data, the illumination data and the traffic flow density data to obtain current environment features. The first acquisition module is configured to acquire a prior confidence probability of the plurality of perception data. The third determination module is configured to determine a conditional probability of the plurality of perception data in the current environment based on the current environment features and a preset environment confidence mapping relationship. The fourth determination module is configured to determine a posterior confidence probability based on a Bayesian formula, the prior confidence probability and the conditional probability. The fifth determination module is configured to determine the fusion weight based on the posterior confidence probability and the plurality of perception data.

[0087] In an optional embodiment, further comprising: The second feature extraction module is configured to perform feature extraction on the plurality of perception data to obtain multi-modal features. The weight determination module is configured to input the posterior confidence probability and the multi-modal features into a neural network model containing an attention mechanism to output the fusion weight.

[0088] In an optional embodiment, further comprising: The time synchronization module is configured to perform time synchronization on the plurality of perception data to obtain a plurality of time-synchronized perception data; the plurality of perception data comprises image data, millimeter wave radar data, laser radar data and vehicle attitude position data. The sixth determination module is configured to determine the fused environment perception data based on the fusion weight and the plurality of perception data.

[0089] In an optional embodiment, the plurality of perception data comprises vehicle motion state data; and further comprising: The model construction module is configured to construct a danger probability prediction model based on the transformer model and the time sequence feature extraction model. The prediction module is configured to input the vehicle motion state data and the fused environment perception data into the danger probability prediction model, and output a danger probability value of a preset time period. The seventh determination module is configured to determine a scene danger level based on the danger probability value.

[0090] In an optional embodiment, the method further comprises: The eighth determination module is configured to determine a low danger level as the scene danger level if the danger probability value is less than a first danger probability threshold. The ninth determination module is configured to determine a medium danger level as the scene danger level if the danger probability value is greater than or equal to the first danger probability threshold and less than or equal to a second danger probability threshold; the second danger probability threshold is greater than the first danger probability threshold. The tenth determination module is configured to determine a high danger level as the scene danger level if the danger probability value is greater than the second danger probability threshold.

[0091] In an optional embodiment, the method further comprises: The first planning module is configured to execute a first driving planning strategy at a first frequency if the scene danger level is a low danger level; the first driving planning strategy comprises generating a candidate path based on the fused environment perception data and a path planning algorithm, and generating an automatic driving instruction based on the candidate path and traffic rule information. The second planning module is configured to execute the first driving planning strategy at a second frequency if the scene danger level is a medium danger level; the second frequency is greater than the first frequency. The third planning module is configured to execute a second driving planning strategy if the scene danger level is a high danger level; the first driving planning strategy comprises determining a target evasive action from a pre-compiled evasive action set based on the fused environment perception data, and generating an automatic driving instruction based on the target evasive action; the pre-compiled evasive action set comprises a plurality of evasive actions; the plurality of evasive actions correspond to the plurality of fused environment perception data in a one-to-one manner.

[0092] The device and method embodiments in the embodiments of the present application are based on the same application concept.

[0093] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal, a server, or a similar computing device. Taking the case of running on a server, Figure 5 is a hardware structure block diagram of a server of a vehicle driving planning method provided in the embodiments of the present application. As shown in Figure 5As shown, the server 500 can vary greatly in configuration and performance, and can include one or more Central Processing Units (CPU) 510 (processor 510 can include, but is not limited to, a microprocessor, a programmable logic device, a processing device, etc.), a memory 530 for storing data, one or more storage media 520 (e.g., one or more mass storage devices) for storing applications 523 or data 522. The memory 530 and the storage media 520 can be of the temporary or persistent storage type. The programs stored in the storage media 520 can include one or more modules, each of which can include a series of instructions for operating on the server. Further, the central processing unit 510 can be configured to communicate with the storage media 520 to execute a series of instructions operating on the server 500. The server 500 can also include one or more power supplies 560, one or more wired or wireless network interfaces 550, one or more input / output interfaces 540, and / or one or more operating systems 521, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0094] The input / output interface 540 can be configured to receive or send data via a network. Examples of the network can include a wireless network provided by a communication provider of the server 500. In one example, the input / output interface 540 includes a network interface controller (NIC) that can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the input / output interface 540 can be a radio frequency (RF) module configured to communicate with the Internet in a wireless manner.

[0095] Those of ordinary skill in the art can understand that, Figure 5 The structure shown is only illustrative and does not limit the structure of the electronic device described above. For example, the server 500 can include more or fewer components than those shown in Figure 5 or have a different configuration than that shown in Figure 5 .

[0096] The embodiments of the present application provide an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the data processing method described above.

[0097] The embodiment of the present application also provides a computer readable storage medium, which can be arranged in a server to store at least one instruction, at least one program, a code set or an instruction set related to a vehicle driving planning method in the method embodiment, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to realize the vehicle driving planning method.

[0098] Optionally, in the embodiment, the storage medium can be located in at least one of the network servers in the computer network. Optionally, in the embodiment, the storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk and various media capable of storing program codes.

[0099] As can be seen from the above embodiments of the vehicle driving planning method, device, electronic equipment or storage medium provided by the present application, in the present application, multi-modal perception data and vehicle environment data are acquired; the multi-modal perception data includes multiple perception data of the vehicle; a fusion weight is determined based on the vehicle environment data and the multiple perception data; the multiple perception data are fused based on the fusion weight to obtain fused environment perception data; a scene danger level is determined based on the multiple perception data and the fused environment perception data; and automatic driving planning is performed based on the scene danger level and the fused environment perception data. In the embodiments of the present application, the multi-modal data fusion weight adapted to the current scene is dynamically generated in combination with the vehicle environment data and the characteristics of each perception data, the multiple perception data are fused according to the fusion weight, and accurate and unified fused environment perception data are output, thereby solving the problems of poor environment adaptability of a single sensor and insufficient robustness of static fusion, improving the accuracy of environment perception in a complex scene; the scene danger level is determined based on the comprehensive analysis of the original perception data and the fused data, the automatic driving strategy adapted to the scene is formulated according to the danger level and the fused perception data, the scene grading decision improves the response efficiency in a sudden scene, reduces the collision risk, and enhances the safety of automatic driving.

[0100] It should be noted that the above-mentioned embodiments of the present application are in the order of description only, and do not represent the advantages and disadvantages of the embodiments. The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be executed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or can be advantageous.

[0101] The various embodiments described in the specification are intended to be exemplary only and the same are not to be taken in a limiting sense. Unless otherwise noted, structures described in this disclosure are not intended to be solely composed of elements as illustrated in the figures and described herein. Rather, unless otherwise specified, structures depicted herein are shown as simplified representations can be comprised of many elements including many elements not specifically shown or described in the figures and / or specification. Furthermore, unless otherwise specified, structures shown in the figures can be implemented with other structures not expressly shown or described. It will be apparent to those skilled in the art that numerous and various embodiments can be derived from the teachings described herein without departing from the scope of the present disclosure.

[0102] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program instructing relevant hardware to complete, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0103] The above description is merely illustrative of the application and is not to be taken in a limiting sense. Any modifications, equivalent replacements, improvements, and the like made within the spirit and principle of the application shall fall within the scope of the application.

Claims

1. A vehicle drive planning method characterized by, The method comprises: acquiring multi-modal perception data and vehicle environment data; the multi-modal perception data comprises a plurality of perception data of a vehicle; determining a fusion weight based on the vehicle environment data and the plurality of perception data; fusing the plurality of perception data based on the fusion weight to obtain fused environment perception data; determining a scene danger level based on the plurality of perception data and the fused environment perception data; performing automatic driving planning based on the scene danger level and the fused environment perception data.

2. The vehicle driving planning method according to claim 1, characterized by, the vehicle environment data comprises weather data, illumination data and traffic flow density data; determining a fusion weight based on the vehicle environment data and the plurality of perception data comprises: extracting features from the weather data, the illumination data and the traffic flow density data to obtain current environment features; acquiring prior confidence probabilities of the plurality of perception data; determining conditional probabilities of the plurality of perception data under the current environment based on the current environment features and a preset environment confidence mapping relationship; determining posterior confidence probabilities based on a Bayesian formula, the prior confidence probabilities and the conditional probabilities; determining the fusion weight based on the posterior confidence probabilities and the plurality of perception data.

3. The vehicle driving planning method according to claim 2, characterized by, determining the fusion weight based on the posterior confidence probabilities and the plurality of perception data comprises: extracting features from the plurality of perception data to obtain multi-modal features; inputting the posterior confidence probabilities and the multi-modal features into a neural network model containing an attention mechanism to output the fusion weight.

4. The vehicle driving planning method according to claim 1, characterized by, fusing the plurality of perception data based on the fusion weight to obtain fused environment perception data comprises: synchronizing the plurality of perception data in time to obtain a plurality of time-synchronized perception data; the plurality of perception data comprises image data, millimeter wave radar data, laser radar data and vehicle attitude and position data; determining the fused environment perception data based on the fusion weight and the plurality of perception data.

5. The vehicle driving planning method according to claim 1, characterized by, the plurality of perception data comprises vehicle motion state data; determining a scene danger level based on the plurality of perception data and the fused environment perception data comprises: constructing a danger probability prediction model based on a transformer model and a time sequence feature extraction model; inputting the vehicle motion state data and the fused environment perception data into the danger probability prediction model to output a danger probability value in a preset time period; determining the scene danger level based on the danger probability value.

6. The vehicle driving planning method according to claim 5, characterized by, determining the scene danger level based on the danger probability value comprises: if the danger probability value is less than a first danger probability threshold, determining a low danger level as the scene danger level; or; if the danger probability value is greater than or equal to the first danger probability threshold and less than or equal to a second danger probability threshold, determining a medium danger level as the scene danger level; the second danger probability threshold is greater than the first danger probability threshold; or; if the danger probability value is greater than the second danger probability threshold, determining a high danger level as the scene danger level.

7. The vehicle driving planning method according to claim 6, characterized by, performing automatic driving planning based on the scene danger level and the fused environment perception data comprises: If the scene danger level is the low danger level, a first driving planning strategy is executed at a first frequency; the first driving planning strategy is to generate a candidate path based on the fused environmental perception data and a path planning algorithm, and to generate an automatic driving instruction based on the candidate path and traffic rule information; or If the scene danger level is the medium danger level, the first driving planning strategy is executed at a second frequency; the second frequency is greater than the first frequency; or If the scene danger level is the high danger level, a second driving planning strategy is executed; the first driving planning strategy is to determine a target evasive action from a pre-compiled evasive action set based on the fused environmental perception data, and to generate the automatic driving instruction based on the target evasive action; the pre-compiled evasive action set includes a plurality of evasive actions; the plurality of evasive actions correspond to a plurality of fused environmental perception data one by one.

8. A vehicle automatic driving device characterized by comprising: The device comprises: an acquisition module configured to acquire multi-modal perception data and vehicle environmental data; the multi-modal perception data includes a plurality of perception data of a vehicle; a first determination module configured to determine a fusion weight based on the vehicle environmental data and the plurality of perception data; a data fusion module configured to fuse the plurality of perception data based on the fusion weight to obtain fused environmental perception data; a second determination module configured to determine a scene danger level based on the plurality of perception data and the fused environmental perception data; a driving planning module configured to perform automatic driving planning based on the scene danger level and the fused environmental perception data.

9. An electronic device, comprising: The electronic device comprises a processor and a memory, the memory stores at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to realize the vehicle driving planning method of any one of claims 1-7.

10. A computer readable storage medium characterized by, The computer readable storage medium stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by the processor to realize the vehicle driving planning method of any one of claims 1-7.

Images