A method and device for predicting intelligent vehicle operation risk fields

Abstract

The present application discloses a method and device for predicting the operation risk field of an intelligent vehicle, which relates to the field of intelligent driving technology. The present application first acquires a bird's-eye view image, a depth map and operation status data based on a first laser radar and a second laser radar, then uses a multi-view feature extraction network to extract multi-view features from the bird's-eye view image and the depth map, and uses a multi-target interactive information extraction network to extract multi-target interactive features from the motion status data, and then fuses the multi-view features and the multi-target interactive features. By using the fused features, real-time evaluation and prediction of operation risks are performed, which can improve the accuracy of risk field prediction and thereby improve the safety and reliability of autonomous driving.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to a method and device for predicting the risk field of intelligent vehicle operation. Background Art

[0002] With the rapid development of autonomous driving technology, the operational safety of intelligent vehicles in complex traffic environments is gaining increasing attention. Traditional risk assessment methods, such as grid map models and artificial potential field methods, while capable of assessing vehicle operational risks to a certain extent, suffer from insufficient prediction accuracy in complex scenarios and an inability to effectively handle multi-vehicle interactions. For example, existing methods struggle to accurately predict the motion state of the target vehicle when handling multi-vehicle interactions, resulting in inaccurate risk assessments and an inability to provide reliable support for autonomous driving decision-making. Summary of the Invention

[0003] The purpose of this application is to provide a method and device for predicting the risk field of intelligent vehicle operation to improve the accuracy of risk field prediction.

[0004] To achieve the above objectives, this application provides the following solutions.

[0005] In a first aspect, the present application provides a method for predicting a risk field for intelligent vehicle operation, comprising:

[0006] Using a first laser radar and a second laser radar installed on the smart car, a bird's-eye view image, a depth map, and operating status data are synchronously acquired at each target time point; each target time point includes a current time point and a preset number of historical time points prior to the current time point; the first laser radar is used to acquire point cloud data representing global map information of the smart car's environment, and the second laser radar is used to acquire point cloud data representing operating status information of the smart car and participating vehicles; the participating vehicles include vehicles within the scanning range of the second laser radar of the smart car;

[0007] A multi-view feature extraction network is used to extract features from the bird's-eye view image and depth image of each target time sequence point to obtain multi-view features of each target time sequence point;

[0008] A multi-target interaction information extraction network is used to extract multi-target interaction features from the motion state data of each target time sequence point, and the multi-target interaction features of each target time sequence point are obtained;

[0009] The multi-view features and multi-target interaction features of each target time point are spliced together to obtain the fusion features of each target time point;

[0010] Based on the fusion features of each target time sequence point, a Bidirectional Long Short-Term Memory (BiLSTM) prediction network is used to obtain the operating status data of the predicted time sequence point; the predicted time sequence point is located after the current time sequence point;

[0011] Based on the operating status data at the predicted time sequence point, the operating risk field of the smart car is calculated.

[0012] In a second aspect, the present application provides a device for predicting a risk field for an intelligent vehicle operation. The device applies the above-mentioned method for predicting a risk field for an intelligent vehicle operation. The device comprises:

[0013] A fusion data acquisition module is configured to utilize a first laser radar and a second laser radar installed on the smart car to synchronously acquire a bird's-eye view image, a depth map, and operating status data for each target time point; each target time point includes a current time point and a preset number of historical time points prior to the current time point; the first laser radar is configured to acquire point cloud data representing global map information of the smart car's environment, and the second laser radar is configured to acquire point cloud data representing operating status information of the smart car and participating vehicles; the participating vehicles include vehicles within the scanning range of the second laser radar of the smart car;

[0014] A multi-view feature extraction module is used to extract features from the bird's-eye view image and depth image of each target time sequence point using a multi-view feature extraction network to obtain multi-view features of each target time sequence point;

[0015] A multi-target interaction feature extraction module is used to extract multi-target interaction features from the motion state data of each target time sequence point using a multi-target interaction information extraction network to obtain multi-target interaction features of each target time sequence point;

[0016] The splicing module is used to splice the multi-view features and multi-target interaction features of each target time point to obtain the fusion features of each target time point;

[0017] A prediction module is used to obtain the operating status data of a predicted time sequence point based on the fusion features of each target time sequence point using a BiLSTM prediction network; the predicted time sequence point is located after the current time sequence point;

[0018] The risk field calculation module is used to calculate the operation risk field of the smart car based on the operation status data at the predicted time sequence point.

[0019] According to the specific embodiments provided in this application, this application has the following technical effects.

[0020] The present application provides a method and device for predicting the operation risk field of an intelligent vehicle. The present application first uses a first laser radar and a second laser radar installed on the intelligent vehicle to synchronously obtain a bird's-eye view, a depth map and operation status data of each target time point, and obtains the bird's-eye view, depth map and operation status data based on the first laser radar and the second laser radar. Then, a multi-view feature extraction network is used to extract multi-view features from the bird's-eye view and the depth map, and a multi-target interactive information extraction network is used to extract multi-target interactive features from the motion status data. Then, the multi-view features and the multi-target interactive features are fused. Real-time evaluation and prediction of operation risks are performed through the fused features, which can improve the accuracy of risk field prediction and thereby improve the safety and reliability of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of a method for predicting risk fields for intelligent vehicle operation provided in one embodiment of the present application.

[0023] Figure 2 A schematic diagram of a method for predicting risk fields for intelligent vehicle operation provided in one embodiment of the present application.

[0024] Figure 3 Schematic diagram of the installation principle of the first laser radar and the second laser radar provided in one embodiment of the present application.

[0025] Figure 4 A flowchart for generating a bird's-eye view image and a depth map is provided in one embodiment of the present application.

[0026] Figure 5 A schematic diagram of operating status data extraction provided in one embodiment of the present application.

[0027] Figure 6 A schematic diagram of the structure of a multi-view feature extraction network provided in one embodiment of the present application.

[0028] Figure 7 A schematic diagram of the structure of a multi-target interactive information extraction network provided in one embodiment of the present application.

[0029] Figure 8 A schematic diagram of the structure of the BiLSTM prediction network provided in one embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0031] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0032] An embodiment of the present application provides a method and device for predicting the risk field for the operation of an intelligent vehicle. The method is a method for predicting the risk field for the operation of an intelligent vehicle based on dual multi-line lidar. By analyzing the characteristics of the movement of objects in urban scenes being affected by the interaction with other surrounding objects and the surrounding environment, integrating the time-series multi-view features of the point cloud and multi-target interaction information, and using artificial potential field theory to establish static and dynamic risk fields for participating vehicles around the vehicle, a feasible solution is provided for the problem of predicting the risk field for the operation of intelligent vehicles in complex scenes, thereby improving the safety and reliability of autonomous driving.

[0033] In an exemplary embodiment, a method for predicting the risk field of intelligent vehicle operation is provided, such as Figure 1 and Figure 2 As shown, the process includes the following steps 101 to 106.

[0034] Step 101, using the first laser radar and the second laser radar installed on the smart car, synchronously obtain the bird's-eye view, depth map and operating status data of each target time point; each target time point includes the current time point and a preset number of historical time points before the current time point; the first laser radar is used to obtain point cloud data representing the global map information of the smart car's environment, and the second laser radar is used to obtain point cloud data representing the operating status information of the smart car and participating vehicles; the participating vehicles include vehicles within the scanning range of the second laser radar of the smart car.

[0035] Step 102: A multi-view feature extraction network is used to extract features from the bird's-eye view image and the depth image of each target time sequence point to obtain multi-view features of each target time sequence point.

[0036] Step 103: A multi-target interaction information extraction network is used to extract multi-target interaction features from the motion state data of each target time sequence point to obtain multi-target interaction features of each target time sequence point.

[0037] In step 104 , the multi-view features and multi-target interaction features of each target time sequence point are respectively spliced to obtain the fusion features of each target time sequence point.

[0038] Step 105: Based on the fusion features of each target time sequence point, a BiLSTM prediction network is used to obtain the operating status data of the predicted time sequence point; the predicted time sequence point is located after the current time sequence point.

[0039] Step 106: Calculate the operation risk field of the smart car based on the operation status data at the predicted time point.

[0040] By implementing steps 101 to 106 above and fusing temporal multi-view features and multi-target interaction features, the motion state of each target vehicle can be more accurately predicted. Compared with traditional models such as fully connected (FC), long short-term memory (LSTM), and bidirectional long short-term memory (BiLSTM), the prediction effect is significantly improved, which can improve the accuracy of risk field prediction and thus enhance the safety and reliability of autonomous driving.

[0041] In another exemplary embodiment, in step 101 above, in order to effectively collect information about the surrounding environment of the smart car and the interaction between participating vehicles in complex traffic scenes, the embodiment of the present application uses dual multi-line laser radars (i.e., the first laser radar and the second laser radar) for environmental perception. The installation position of the dual multi-line laser radar system is as follows: Figure 3 As shown, the LiDAR is mounted vertically. The upper LiDAR (the first LiDAR) is used to acquire global environmental map information in real time, generating a time-series bird's-eye view and forward depth map. The lower LiDAR is used to monitor the operating status of participating vehicles in real time, capturing complex interaction information between the smart car and participating vehicles.

[0042] The installation height and installation angle of the first laser radar and the second laser radar satisfy the following formula:

[0043] (1)

[0044] in, is the maximum projection distance of the first laser radar’s scanning line on the ground, is the minimum projection distance of the first laser radar’s scanning line on the ground, is the installation height of the first radar, is the angle between the horizontal line and the lowest scanning line of the first laser radar, representing the installation angle. is the vertical angular resolution of the first lidar, is the i-th scanning line of the first laser radar, representing the corresponding scan line of the first laser radar; is the maximum projection distance of the second laser radar’s scanning line on the ground, is the minimum projection distance of the second laser radar’s scanning line on the ground, is the installation height of the second radar, is the angle between the horizontal line and the lowest scanning line of the second laser radar, representing the installation angle. is the vertical angular resolution of the second lidar, is the i-th scanning line of the second laser radar, representing The corresponding scan line of the second lidar.

[0045] In another exemplary embodiment, the above step 101 may be replaced by the following steps 201 to 204 .

[0046] Step 201: Calculate the time synchronization error between the first laser radar and the second laser radar.

[0047] Step 202: Control the first laser radar and the second laser radar respectively according to the time synchronization error to synchronously acquire the first laser radar point cloud data and the second laser radar point cloud data of each target time sequence point.

[0048] Step 203: Convert the first lidar point cloud data of each target time sequence point into a bird's-eye view map and a depth map.

[0049] Step 204 , extract motion features from the second lidar point cloud data of each target time sequence point to obtain the operating status data of each target time sequence point; the operating status data includes: the operating status vector of the smart car and the operating status vectors of each participating vehicle.

[0050] In another exemplary embodiment, the above-mentioned step 203 determines the time synchronization error by solving the synchronization error function, specifically including: constructing a spatiotemporal joint compensation model and structural feature constraints of the first lidar and the second lidar; constructing a synchronization error function based on the spatiotemporal joint compensation model and structural feature constraints; optimizing and solving the synchronization error function to obtain the time synchronization error.

[0051] The time synchronization error between the first laser radar and the second laser radar is defined as , point cloud coordinates affected by laser velocity The impact produces offset, and the spatiotemporal joint compensation model is determined as:

[0052]

[0053] in, The timing points obtained for the second lidar scan The 3D point cloud of the mth calibration point in the second lidar point cloud data, The time points obtained for the first lidar scan The 3D point cloud of the mth calibration point in the first lidar point cloud data, is the laser linear velocity, v∈ , calculated by the adjacent frame point cloud: ; Obtained for the first lidar scan The 3D point cloud of the mth calibration point in the first lidar data of the frame, Obtained for the first lidar scan The 3D point cloud of the mth calibration point in the first lidar data of the frame, where N is the number of calibration points. is the time interval between two adjacent frames, is the rotation matrix of the second lidar coordinate system relative to the first lidar coordinate system, is the translation vector of the second lidar coordinate system relative to the first lidar coordinate system, is the time synchronization error between the first lidar and the second lidar (unit: seconds).

[0054] The stable structures of planes and straight lines in the environment are selected as constraint planes and constraint lines respectively, and plane constraints and edge constraints are established.

[0055] Plane constraint: constrain the plane , its normal vector n should satisfy:

[0056]

[0057] in, is the normal vector of the constraint plane, with superscript T represents transpose, is the distance between the constraining plane of the mth calibration point and the origin of the first lidar coordinate system.

[0058] Edge Constraint: Constrain Line , the direction vector u satisfies:

[0059]

[0060] in, Timing points obtained for the second lidar scan The direction vector of the constraint line in the second lidar point cloud data, The time points obtained for the first lidar scan The direction vector of the constraining line in the first lidar point cloud data.

[0061] Combining the point distance error and structural constraints, a differentially optimized synchronization error function is constructed:

[0062]

[0063]

[0064]

[0065] in, is the synchronization error function, is the rotation matrix of the second lidar coordinate system relative to the first lidar coordinate system, is the translation vector of the second lidar coordinate system relative to the first lidar coordinate system, is the time synchronization error between the first laser radar and the second laser radar, is the point cloud registration error function, is the plane constraint error function; Timing points obtained for the second lidar scan The 3D point cloud of the mth calibration point in the second lidar point cloud data, The time points obtained for the first lidar scan The 3D point cloud of the mth calibration point in the first lidar point cloud data, is the laser linear velocity, is the weight of the 3D point cloud of the mth calibration point; is the plane constraint error weight, is the normal vector of the constraint plane, with superscript T represents transpose, is the distance between the constraining plane of the mth calibration point and the origin of the first lidar coordinate system.

[0066] In another exemplary embodiment, the synchronization error function may be solved using a Lie algebra parameterization method, but the invention is not limited to this solution method.

[0067] In another exemplary embodiment, the above step 203 converts the original point cloud of the first lidar (i.e., the first lidar point cloud data) into image forms such as a bird's-eye view and a front-view depth map based on the point cloud multi-view method for processing, thereby providing the necessary high-precision environmental perception information input for subsequent motion target state prediction.

[0068] In the embodiment of the present application, in order to convert the original point cloud of the vehicle-mounted laser radar into a bird's-eye view and a front-view depth map, the Figure 4 The multi-view generation process shown is obtained.

[0069] The original point cloud of the 3D LiDAR is projected into a 2D image and transformed according to the internal parameters of the corresponding camera to obtain a bird's-eye view and depth map. The conversion relationship from the LiDAR coordinate system to the image coordinate system is:

[0070]

[0071] Where, is the pixel coordinate ， is the laser radar coordinate, represents the rotation matrix from the lidar coordinate system to the camera coordinate system, represents the three-dimensional translation vector from the lidar coordinate system to the world coordinate system, 、 is the camera's internal parameter, is the image resolution, is the camera focal length. is the pixel coordinate The corresponding depth value.

[0072] The camera involved in the conversion process is a virtual camera, that is, a virtual camera is set to shoot in the downward and forward directions to obtain a bird's-eye view and a depth map.

[0073] In another exemplary embodiment, target vehicle state prediction in complex traffic scenarios involves estimating future states based on the vehicle's own operating state and the temporal and spatial interactions with its surrounding environment. To effectively exploit the interactive features in complex traffic data, the input data sensed by the lower second LiDAR should include vehicle dimensions and operating state parameters.

[0074] Complex traffic scene multi-target information interaction scene such as Figure 5 As shown in the figure, the smart car equipped with a lidar sensor is located in the center of the road, and its operating state model is:

[0075] (3)

[0076] Where, is the running state vector of the smart car, L is the length of the smart car; W is the width of the smart car; 、 are the horizontal and vertical coordinates of the center point of the second laser radar at time t; is the instantaneous speed of the smart car at time t; It is the historical status data of smart car operation. 、 are the horizontal and vertical coordinate values of the center point of the second laser radar at time t-5, is the instantaneous speed of the smart car at time t-5, 、 are the horizontal and vertical coordinates of the center point of the second laser radar at time t-1, is the instantaneous speed of the smart car at time t-1.

[0077] The vehicle in front of the smart car is sensed by the second lidar , right front vehicle , vehicle in front of left Vehicle behind right As a participating vehicle whose state behavior needs to be predicted. For example, its running state model is:

[0078] (4)

[0079] Where, For the vehicle directly ahead The running state vector, For the vehicle directly ahead length; For the vehicle directly ahead Width; 、 is the vehicle directly in front at time t The horizontal and vertical coordinate values of the center point; is the vehicle directly in front at time t The instantaneous speed of For the vehicle directly ahead Historical status data, 、 The vehicle directly in front at time t-5 The horizontal and vertical coordinate values of the center point, The vehicle directly in front at time t-5 The instantaneous speed, 、 is the vehicle directly in front at time t-1 The horizontal and vertical coordinate values of the center point, is the vehicle directly in front at time t-1 The instantaneous speed.

[0080] Smart car and the vehicle in front , right front vehicle , vehicle in front of left Vehicle behind right Operation status data of complex traffic scenarios for:

[0081] (5)

[0082] In the formula is the running state vector of the smart car, They are the operating state vectors of the four participating vehicles, namely the vehicle in front , right front vehicle , vehicle in front of left Vehicle behind right The running state vector.

[0083] The overall network architecture of the multi-view feature extraction network, multi-target interactive information extraction network and BiLSTM prediction network proposed in the embodiment of the present application is as follows: Figure 2 As shown in the figure, by inputting the historical operating status data of multiple targets in complex traffic scenes obtained by sensors such as lidar and the corresponding time-series bird's-eye view and depth map, the prediction network outputs the operating status data of multiple targets at future moments, thereby realizing the risk field prediction at future moments.

[0084] In another exemplary embodiment, the multi-view feature extraction network uses two improved VGG19 (Visual Geometry Group 19) networks to extract multi-view features from point clouds. One branch extracts depth map features, and the other extracts bird's-eye view features. The improved VGG19 network adds several convolutional layers to a shallow CNN (Convolutional Neural Networks). Because adding convolutional layers is more conducive to image feature extraction than adding fully connected layers, the improved VGG19 network is more likely to overcome the lack of diversity and complexity in traffic scenes than a shallow convolutional neural network, ultimately achieving better spatiotemporal feature extraction results. Figure 6 As shown, the input layer image size of the multi-view feature extraction network provided by the embodiment of the present application is 224×224 pixels, and the number of channels is 3; both VGG19 networks have 16 convolutional layers. Among them, the maximum pooling layer is followed by the 2nd, 4th, 8th, 12th, and 16th convolutional layers. The size of the convolution kernel in the convolutional layer is gradually reduced by half from 224×224 to 14×14. The use of a gradually decreasing convolution kernel is equivalent to adding implicit regularization, which can improve the feature extraction ability of the network and increase the computing speed of the network. The two VGG19s have 4 fully connected layers with 4096, 4096, 1000, and 5 neurons respectively. The last layer splices the depth map features and bird's-eye view features of the corresponding time points, and extracts high-level abstract combination features in the multi-view scene to obtain multi-view features.

[0085] In another exemplary embodiment, the multi-target interaction relationship corresponds to a one-dimensional time series. The embodiment of the present application uses a one-dimensional CNN network to extract potential interaction relationships. Specifically, the corresponding local information is calculated by sliding a convolution kernel of a specific size over the local area of the input data in sequence.

[0086] like Figure 7 As shown, the multi-target interaction information extraction network takes as input the motion state data of each target at a time point in a complex traffic scene acquired by a second lidar. The output is the potential multi-target interaction features extracted at those time points by a one-dimensional CNN network. These multi-target interaction features are represented by representing each dynamic target (smart car or participating vehicle) as a node. The nodes corresponding to any two dynamic targets in the same point cloud frame are connected by a solid line to represent a spatial edge, while the same target in adjacent frames is connected by a dashed line to represent a temporal edge.

[0087] In another exemplary embodiment, the output features of the multi-view feature extraction network and the multi-target interactive information extraction network are fused and input into the established BiLSTM prediction network for running state prediction. Figure 8 As shown in the figure, the BiLSTM prediction network consists of a BiLSTM layer, an Attention layer, and an output layer.

[0088] In another exemplary embodiment, in order to quickly quantify the driving risk level of a vehicle in an environment, the embodiment of the present application establishes an operating risk field for each participating vehicle. The operating risk field comprehensively considers the external dimensions of the participating vehicle and the relative motion between the participating vehicle and the smart car. The operating risk field is the sum of the static risk field and the dynamic risk field. The calculation formula of the operating risk field is:

[0089] (6)

[0090] (7)

[0091] (8)

[0092] (9)

[0093] (10)

[0094] in, for The operational risk field of the smart car for the jth participating vehicle at the moment, for The static risk field of the smart car with respect to the jth participating vehicle at time, for The dynamic risk field of the smart car with respect to the jth participating vehicle at time instant;

[0095] is the field strength coefficient, B and C are the first intermediate function and the second intermediate function respectively, β are high-order coefficients, and for The shape parameters of the jth participating vehicle at time, and They are The lateral size coefficient and longitudinal size coefficient of the jth participating vehicle at the moment, and They are The length and width of the jth participating vehicle at time;

[0096] for The speed relationship parameter between the jth parameter vehicle and the smart car at the moment, for The jth parameter at the moment is the direction relationship parameter between the vehicle and the smart car, and They are The horizontal and vertical coordinates of the smart car on the road at a given moment, where the horizontal coordinate is parallel to the road and the vertical coordinate is perpendicular to the road. for Relative velocity coefficient at time, for Moment speed coefficient, for The speed of the jth participating vehicle at time, for The speed of smart cars at all times.

[0097] for The heading angle of the jth participating vehicle on the road at time, and They are The horizontal and vertical coordinates of the jth participating vehicle on the road at time.

[0098] According to the specific embodiments provided in this application, this application has the following technical effects.

[0099] High prediction accuracy: By fusing point cloud temporal multi-view features and multi-target interaction information, the motion state of the target vehicle can be predicted more accurately. Compared with traditional FC, LSTM, BiLSTM and other models, the prediction effect is significantly improved.

[0100] Comprehensive risk assessment: The static and dynamic risk field models established based on artificial potential field theory comprehensively consider factors such as the size and relative motion of obstacles. They can comprehensively and real-timely assess the operating risks of smart cars, providing a more reliable basis for autonomous driving decisions.

[0101] Strong adaptability: It is suitable for complex traffic scenarios, can effectively handle multi-vehicle interactions, different road conditions, etc., and has strong environmental adaptability.

[0102] Improved safety: By predicting and assessing operational risks in advance, it provides support for smart car decisions such as obstacle avoidance and path planning, helping to reduce the incidence of traffic accidents and improve the safety of autonomous driving.

[0103] Based on the same inventive concept, embodiments of the present application also provide an intelligent vehicle operation risk field prediction device for implementing the aforementioned intelligent vehicle operation risk field prediction method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the intelligent vehicle operation risk field prediction device provided below can be found in the limitations of the intelligent vehicle operation risk field prediction method described above and will not be further elaborated here.

[0104] In an exemplary embodiment, a device for predicting a risk field for an intelligent vehicle operation is provided, comprising:

[0105] A fusion data acquisition module is used to use the first laser radar and the second laser radar installed on the smart car to synchronously obtain the bird's-eye view, depth map and operating status data of each target time point; each target time point includes the current time point and a preset number of historical time points before the current time point; the first laser radar is used to obtain point cloud data representing the global map information of the smart car's environment, and the second laser radar is used to obtain point cloud data representing the operating status information of the smart car and participating vehicles; the participating vehicles include vehicles within the scanning range of the second laser radar of the smart car.

[0106] The multi-view feature extraction module is used to extract features from the bird's-eye view image and depth image of each target time sequence point using a multi-view feature extraction network to obtain the multi-view features of each target time sequence point.

[0107] The multi-target interaction feature extraction module is used to extract the multi-target interaction features of the motion state data of each target time sequence point using a multi-target interaction information extraction network to obtain the multi-target interaction features of each target time sequence point.

[0108] The splicing module is used to splice the multi-view features and multi-target interaction features of each target time point respectively to obtain the fusion features of each target time point.

[0109] A prediction module is used to obtain the operating status data of a predicted time sequence point based on the fusion features of each target time sequence point using a BiLSTM prediction network; the predicted time sequence point is located after the current time sequence point;

[0110] The risk field calculation module is used to calculate the operation risk field of the smart car based on the operation status data at the predicted time sequence point.

[0111] In another exemplary embodiment, the fusion data acquisition module includes: a dual multi-line laser radar environment perception module, a point cloud multi-view generation module, and a multi-target information interaction network module.

[0112] Dual multi-line lidar environment perception module: used to obtain the interaction information between the environment around the smart car and traffic participants.

[0113] Point cloud multi-view generation module: used to convert raw point cloud data into bird's-eye view and depth map.

[0114] Multi-target information interaction network module: used to extract motion status data of multiple targets in complex traffic scenes.

[0115] In another exemplary embodiment, the aforementioned multi-view feature extraction module, multi-target interaction feature extraction module, splicing module, prediction module, and risk field calculation module constitute a vehicle operation risk prediction module. This vehicle operation risk prediction module includes a multi-view feature extraction network, a multi-target interaction information extraction network, a BiLSTM prediction network, and an operation risk assessment network, and is used to predict the motion state of a target vehicle and assess the operation risk of the smart car.

[0116] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for predicting the risk field of intelligent vehicle operation, characterized in that: include: Using a first laser radar and a second laser radar installed on the smart car, a bird's-eye view image, a depth map, and operating status data are synchronously acquired at each target time point; each target time point includes a current time point and a preset number of historical time points prior to the current time point; the first laser radar is used to acquire point cloud data representing global map information of the smart car's environment, and the second laser radar is used to acquire point cloud data representing operating status information of the smart car and participating vehicles; the participating vehicles include vehicles within the scanning range of the second laser radar of the smart car; A multi-view feature extraction network is used to extract features from the bird's-eye view image and depth image of each target time sequence point to obtain multi-view features of each target time sequence point; A multi-target interaction information extraction network is used to extract multi-target interaction features from the motion state data of each target time sequence point, and the multi-target interaction features of each target time sequence point are obtained; The multi-view features and multi-target interaction features of each target time point are spliced together to obtain the fusion features of each target time point; According to the fusion features of each target time sequence point, a BiLSTM prediction network is used to obtain the operating status data of the predicted time sequence point; the predicted time sequence point is located after the current time sequence point; Calculating the operation risk field of the smart car based on the operation status data at the predicted time sequence point; Using the first and second LiDARs installed on the smart car, the bird's-eye view, depth map, and operating status data of each target time point are synchronously acquired, including: Solve the time synchronization error between the first laser radar and the second laser radar; According to the time synchronization error, the first laser radar and the second laser radar are controlled respectively to synchronously obtain the first laser radar point cloud data and the second laser radar point cloud data of each target time sequence point; Convert the first lidar point cloud data of each target time sequence point into a bird's-eye view image and a depth map; Extract motion features from the second laser radar point cloud data at each target time sequence point to obtain operating state data for each target time sequence point; the operating state data includes: an operating state vector of the smart car and an operating state vector of each participating vehicle; Solving the time synchronization error between the first lidar and the second lidar includes: Constructing a spatiotemporal joint compensation model and structural feature constraints for the first and second lidars; constructing a synchronization error function according to the spatiotemporal joint compensation model and structural feature constraints; Optimizing and solving the synchronization error function to obtain the time synchronization error; The spatiotemporal joint compensation model is: ； in, The timing points obtained for the second lidar scan The 3D point cloud of the mth calibration point in the second lidar point cloud data, The time points obtained for the first lidar scan The 3D point cloud of the mth calibration point in the first lidar point cloud data, is the laser linear velocity, is the rotation matrix of the second lidar coordinate system relative to the first lidar coordinate system, is the translation vector of the second lidar coordinate system relative to the first lidar coordinate system, is the time synchronization error between the first laser radar and the second laser radar; The structural feature constraints include plane constraints and edge constraints; The plane constraints are: ； in, Timing points obtained for the second lidar scan The 3D point cloud of the mth calibration point in the second lidar point cloud data, is the laser linear velocity, is the rotation matrix of the second lidar coordinate system relative to the first lidar coordinate system, is the translation vector of the second lidar coordinate system relative to the first lidar coordinate system, is the time synchronization error between the first laser radar and the second laser radar, is the normal vector of the constraint plane, and the superscript T represents the transpose. is the distance between the constraining plane of the mth calibration point and the origin of the first lidar coordinate system; The edge constraints are: ； in, Timing points obtained for the second lidar scan The direction vector of the constraint line in the second lidar point cloud data, The time points obtained for the first lidar scan A direction vector of the constraint line in the first lidar point cloud data; The synchronization error function is: ； ； ； in, is the synchronization error function, is the rotation matrix of the second lidar coordinate system relative to the first lidar coordinate system, is the translation vector of the second lidar coordinate system relative to the first lidar coordinate system, is the time synchronization error between the first laser radar and the second laser radar, is the point cloud registration error function, is the plane constraint error function; The timing points obtained for the second lidar scan The 3D point cloud of the mth calibration point in the second lidar point cloud data, The time points obtained for the first lidar scan The 3D point cloud of the mth calibration point in the first lidar point cloud data, is the laser linear velocity, is the weight of the 3D point cloud of the mth calibration point; is the plane constraint error weight, is the normal vector of the constraint plane, and the superscript T represents the transpose. is the distance between the constraining plane of the mth calibration point and the origin of the first lidar coordinate system.

2. The method for predicting the operation risk field of an intelligent vehicle according to claim 1, characterized in that: The first laser radar and the second laser radar are both installed on the top of the smart car, with the first laser radar located above the second laser radar. The installation heights and installation angles of the first laser radar and the second laser radar satisfy the following formula: ； in, is the maximum projection distance of the first laser radar’s scanning line on the ground, is the minimum projection distance of the first laser radar’s scanning line on the ground, is the installation height of the first radar, is the angle between the horizontal line and the lowest scanning line of the first laser radar, representing the installation angle. is the vertical angular resolution of the first lidar, is the i-th scanning line of the first laser radar, representing the corresponding scan line of the first laser radar; is the maximum projection distance of the second laser radar’s scanning line on the ground, is the minimum projection distance of the second laser radar’s scanning line on the ground, is the installation height of the second radar, is the angle between the horizontal line and the lowest scanning line of the second laser radar, representing the installation angle. is the vertical angular resolution of the second lidar, is the i-th scanning line of the second laser radar, representing The corresponding scan line of the second lidar.

3. The method for predicting the operation risk field of an intelligent vehicle according to claim 1, characterized in that: The multi-view feature extraction network includes: a first VGG19 network, a second VGG19 network and a splicing network; The first VGG19 network is used to extract features from the bird's-eye view image to obtain bird's-eye view features; The second VGG19 network is used to extract features from the depth map to obtain depth map features; The stitching network is used to connect the bird's-eye view features and the depth map features to obtain multi-view features.

4. The method for predicting the operation risk field of an intelligent vehicle according to claim 3, characterized in that: The first VGG19 network and the second VGG19 network are both obtained by adding convolutional layers to the CNN network; The first VGG19 network and the second VGG19 network each include an input layer, 16 convolutional layers, 5 maximum pooling layers, a flattening layer, and 4 fully connected layers; The 16 convolutional layers are connected in sequence, the 5 pooling layers are arranged after the 2nd, 4th, 8th, 12th and 16th convolutional layers respectively, and the input layer is arranged before the 1st convolutional layer; The maximum pooling layer located after the 16th convolutional layer is connected to the flattening layer, and the flattening layer is sequentially connected to four fully connected layers, wherein the last fully connected layer serves as the output layer, and the output layer is connected to the splicing network.

5. A device for predicting risk fields in intelligent vehicle operation, characterized in that: The intelligent vehicle operation risk field prediction device applies the intelligent vehicle operation risk field prediction method according to any one of claims 1 to 4, and the intelligent vehicle operation risk field prediction device includes: A fusion data acquisition module is configured to utilize a first laser radar and a second laser radar installed on the smart car to synchronously acquire a bird's-eye view image, a depth map, and operating status data for each target time point; each target time point includes a current time point and a preset number of historical time points prior to the current time point; the first laser radar is configured to acquire point cloud data representing global map information of the smart car's environment, and the second laser radar is configured to acquire point cloud data representing operating status information of the smart car and participating vehicles; the participating vehicles include vehicles within the scanning range of the second laser radar of the smart car; A multi-view feature extraction module is used to extract features from the bird's-eye view image and depth image of each target time sequence point using a multi-view feature extraction network to obtain multi-view features of each target time sequence point; A multi-target interaction feature extraction module is used to extract multi-target interaction features from the motion state data of each target time sequence point using a multi-target interaction information extraction network to obtain multi-target interaction features of each target time sequence point; The splicing module is used to splice the multi-view features and multi-target interaction features of each target time point to obtain the fusion features of each target time point; A prediction module is used to obtain the operating status data of a predicted time sequence point based on the fusion features of each target time sequence point using a BiLSTM prediction network; the predicted time sequence point is located after the current time sequence point; The risk field calculation module is used to calculate the operation risk field of the smart car based on the operation status data at the predicted time sequence point.

Images