Intelligent automobile operation risk field prediction method and device

Abstract

The invention discloses an intelligent automobile operation risk field prediction method and device, and relates to the technical field of intelligent driving. Firstly, based on a first laser radar and a second laser radar, a bird's-eye view, a depth map and operation state data are obtained; then performing multi-view feature extraction on the aerial view and the depth map by adopting a multi-view feature extraction network, performing multi-target interaction feature extraction on the motion state data by adopting a multi-target interaction information extraction network, and then performing fusion of multi-view features and multi-target interaction features; the real-time evaluation and prediction of the operation risk are carried out through the fusion features, the prediction precision of the risk field can be improved, and then the safety and reliability of automatic driving are improved.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and particularly to a method and device for predicting the operation risk field of an intelligent vehicle. Background Art

[0002] With the rapid development of autonomous driving technology, the operation safety problem of intelligent vehicles in complex traffic environments has attracted increasing attention. Traditional risk assessment methods, such as grid map models and artificial potential field methods, although can evaluate the operation risk of vehicles to a certain extent, have problems such as insufficient prediction accuracy and inability to effectively handle multi-vehicle interactions in complex scenarios. For example, when dealing with multi-vehicle interactions, existing methods are difficult to accurately predict the motion state of the target vehicle, resulting in inaccurate risk assessment and unable to provide reliable support for autonomous driving decisions. Summary of the Invention

[0003] The purpose of this application is to provide a method and device for predicting the operation risk field of an intelligent vehicle to improve the accuracy of risk field prediction.

[0004] To achieve the above purpose, the following solutions are provided in this application.

[0005] In the first aspect, this application provides a method for predicting the operation risk field of an intelligent vehicle, including: Using the first lidar and the second lidar installed on the intelligent vehicle, synchronously obtain the bird's-eye view, depth map, and operation state data of each target time sequence point; each target time sequence point includes the current time sequence point and a preset number of historical time sequence points before the current time sequence point; the first lidar is used to obtain point cloud data representing the global map information of the environment of the intelligent vehicle, and the second lidar is used to obtain point cloud data representing the operation state information of the intelligent vehicle and the participating vehicles; the participating vehicles include the vehicles within the scanning range of the second lidar of the intelligent vehicle; Adopt a multi-view feature extraction network to extract features from the bird's-eye view and depth map of each target time sequence point to obtain the multi-view features of each target time sequence point; Adopt a multi-object interaction information extraction network to extract multi-object interaction features from the motion state data of each target time sequence point to obtain the multi-object interaction features of each target time sequence point; Respectively splice the multi-view features and multi-object interaction features of each target time sequence point to obtain the fusion features of each target time sequence point; According to the fusion features of each target time sequence point, adopt a BiLSTM (Bidirectional Long Short-Term Memory) prediction network to obtain the operation state data of the predicted time sequence point; the predicted time sequence point is after the current time sequence point; Based on the operation state data at the predicted time sequence points, calculate the operation risk field of the intelligent vehicle.

[0006] In a second aspect, the present application provides an intelligent vehicle operation risk field prediction device. The intelligent vehicle operation risk field prediction device applies the above-mentioned intelligent vehicle operation risk field prediction method. The intelligent vehicle operation risk field prediction device includes: A fusion data acquisition module, configured to synchronously acquire the bird's-eye view, depth map, and operation state data of each target time sequence point by using the first lidar and the second lidar installed on the intelligent vehicle; each target time sequence point includes the current time sequence point and a preset number of historical time sequence points before the current time sequence point; the first lidar is used to acquire point cloud data representing the environmental global map information of the intelligent vehicle, and the second lidar is used to acquire point cloud data representing the operation state information of the intelligent vehicle and the participating vehicles; the participating vehicles include the vehicles within the scanning range of the second lidar of the intelligent vehicle; A multi-view feature extraction module, configured to perform feature extraction on the bird's-eye view and depth map of each target time sequence point by using a multi-view feature extraction network to obtain the multi-view features of each target time sequence point; A multi-target interaction feature extraction module, configured to perform multi-target interaction feature extraction on the motion state data of each target time sequence point by using a multi-target interaction information extraction network to obtain the multi-target interaction features of each target time sequence point; A splicing module, configured to splice the multi-view features and multi-target interaction features of each target time sequence point respectively to obtain the fusion features of each target time sequence point; A prediction module, configured to obtain the operation state data of the predicted time sequence point by using a BiLSTM prediction network according to the fusion features of each target time sequence point; the predicted time sequence point is located after the current time sequence point; A risk field calculation module, configured to calculate the operation risk field of the intelligent vehicle based on the operation state data of the predicted time sequence point.

[0007] According to the specific embodiments provided by the present application, the present application has the following technical effects.

[0008] The present application provides a method and device for predicting the operation risk field of an intelligent vehicle. First, the present application uses a first lidar and a second lidar installed on the intelligent vehicle to synchronously obtain the bird's-eye view, depth map, and operation status data at each target time point. Based on the first lidar and the second lidar, the bird's-eye view, depth map, and operation status data are obtained. Then, a multi-view feature extraction network is used to extract multi-view features from the bird's-eye view and the depth map, and a multi-object interaction information extraction network is used to extract multi-object interaction features from the motion status data. Then, the multi-view features and the multi-object interaction features are fused, and the real-time evaluation and prediction of the operation risk are performed through the fused features, which can improve the accuracy of the risk field prediction, and further improve the safety and reliability of autonomous driving. Description of the Drawings

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0010] Figure 1 It is a schematic flowchart of a method for predicting the operation risk field of an intelligent vehicle provided by an embodiment of the present application.

[0011] Figure 2 It is a schematic diagram of the principle of a method for predicting the operation risk field of an intelligent vehicle provided by an embodiment of the present application.

[0012] Figure 3 It is a schematic installation principle diagram of the first lidar and the second lidar provided by an embodiment of the present application.

[0013] Figure 4 It is a flowchart of generating a bird's-eye view and a depth map provided by an embodiment of the present application.

[0014] Figure 5 It is a schematic diagram of extracting operation status data provided by an embodiment of the present application.

[0015] Figure 6 It is a schematic structural diagram of a multi-view feature extraction network provided by an embodiment of the present application.

[0016] Figure 7 It is a schematic structural diagram of a multi-object interaction information extraction network provided by an embodiment of the present application.

[0017] Figure 8 It is a schematic structural diagram of a BiLSTM prediction network provided by an embodiment of the present application. Detailed Embodiments

[0018] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0019] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0020] The embodiments of the present application provide an intelligent vehicle operation risk field prediction method and device. The method is an intelligent vehicle operation risk field prediction method based on dual multi-line lidar. By analyzing the characteristics of object movement affected by the interaction of other surrounding objects and the surrounding environment in the urban scene, fusing the multi-view features of point cloud time series and multi-object interaction information, and using the artificial potential field theory to establish the static and dynamic risk fields of the vehicles participating in the vehicle around, it provides a feasible solution for the problem of intelligent vehicle operation risk field prediction in complex scenarios, so as to improve the safety and reliability of autonomous driving.

[0021] In an exemplary embodiment, an intelligent vehicle operation risk field prediction method is provided, as Figure 1 and Figure 2 shown, including the following steps 101-step 106.

[0022] Step 101, use the first lidar and the second lidar installed on the intelligent vehicle to synchronously obtain the bird's-eye view, depth map, and operation state data of each target time series point; each target time series point includes the current time series point and a preset number of historical time series points before the current time series point; the first lidar is used to obtain point cloud data representing the environmental global map information of the intelligent vehicle, and the second lidar is used to obtain point cloud data representing the operation state information of the intelligent vehicle and the participating vehicles; the participating vehicles include the vehicles within the scanning range of the second lidar of the intelligent vehicle.

[0023] Step 102, use a multi-view feature extraction network to extract features from the bird's-eye view and depth map of each target time series point to obtain the multi-view features of each target time series point.

[0024] Step 103, use a multi-object interaction information extraction network to extract multi-object interaction features from the motion state data of each target time series point to obtain the multi-object interaction features of each target time series point.

[0025] Step 104, splice the multi-view features and multi-object interaction features of each target time series point respectively to obtain the fusion features of each target time series point.

[0026] Step 105: According to the fusion features of each target time point, use a BiLSTM prediction network to obtain the operating state data of the predicted time point; the predicted time point is located after the current time point.

[0027] Step 106: Calculate the operating risk field of the intelligent vehicle based on the operating state data of the predicted time point.

[0028] Implementing the above Steps 101 - 106, by fusing time - series multi - view features and multi - target interaction features, the motion states of each target vehicle can be predicted more accurately. Compared with traditional models such as Fully Connected (FC), Long Short - Term Memory (LSTM), and Bidirectional Long Short - Term Memory (BiLSTM), the prediction effect is significantly improved, the accuracy of risk field prediction can be improved, and thus the safety and reliability of autonomous driving can be enhanced.

[0029] In another exemplary embodiment, in the above Step 101, in order to effectively collect the environmental information around the intelligent vehicle and the interaction information between participating vehicles in a complex traffic scenario, the embodiment of the present application uses a dual multi - line lidar (i.e., the first lidar and the second lidar) for environmental perception. The installation positions of the dual multi - line lidar system are as Figure 3 shown. The lidar is installed vertically. The upper - side lidar (the first lidar) is used to obtain the global environmental map information in real time for generating a time - series top - down bird's - eye view and a time - series front - view depth map. The lower - side lidar is used to detect the operating state data of the participating vehicles in real time to obtain the complex interaction information between the intelligent vehicle and the participating vehicles.

[0030] The installation heights and installation angles of the first lidar and the second lidar satisfy the following formula: (1) Where is the maximum projection distance of the scanning line of the first lidar on the ground, is the minimum projection distance of the scanning line of the first lidar on the ground, is the installation height of the first lidar, is the angle between the horizontal line and the lowermost scanning line of the first lidar, representing the installation angle, is the vertical angular resolution of the first lidar, is the i - th scanning line of the first lidar, representing the corresponding scanning line of the first lidar; is the maximum projection distance of the scanning line of the second lidar on the ground, is the minimum projection distance of the scanning line of the second lidar on the ground, is the installation height of the second radar, is the angle between the horizontal line and the lowermost scanning line of the second lidar, representing the installation angle, is the vertical angular resolution of the second lidar, is the i-th scanning line of the second lidar, representing the corresponding scanning line of the second lidar.

[0031] In another exemplary embodiment, the above step 101 may be replaced by the following steps 201 - 204.

[0032] Step 201, solve the time synchronization error between the first lidar and the second lidar.

[0033] Step 202, according to the time synchronization error, control the first lidar and the second lidar respectively to synchronously obtain the first lidar point cloud data and the second lidar point cloud data of each target time sequence point.

[0034] Step 203, convert the first lidar point cloud data of each target time sequence point into a bird's-eye view and a depth map.

[0035] Step 204, perform motion feature extraction on the second lidar point cloud data of each target time sequence point to obtain the running state data of each target time sequence point; the running state data includes: the running state vector of the intelligent vehicle and the running state vectors of each participating vehicle.

[0036] In another exemplary embodiment, the above step 203 determines the time synchronization error by solving a synchronization error function, specifically including: constructing a spatio-temporal joint compensation model and a structural feature constraint for the first lidar and the second lidar; constructing a synchronization error function according to the spatio-temporal joint compensation model and the structural feature constraint; and performing optimization solution on the synchronization error function to obtain the time synchronization error.

[0037] Define the time synchronization error between the first lidar and the second lidar as , the point cloud coordinates are affected by the laser speed and generate an offset, and determine the spatio-temporal joint compensation model as:

[0038] where, is the time sequence point obtained by the second lidar scanning the three-dimensional point cloud of the m-th calibration point in the second lidar point cloud data of The time-series points obtained from the first lidar scan The 3D point cloud of the m-th calibrated point in the first lidar point cloud data is the lidar linear velocity v∈ , calculated from adjacent frame point clouds ; is the 3D point cloud of the m-th calibrated point in the first lidar data of the frame obtained from the first lidar scan is the 3D point cloud of the m-th calibrated point in the first lidar data of the frame obtained from the first lidar scan, N is the number of calibrated points is the time interval between two adjacent frames is the rotation matrix of the second lidar coordinate system relative to the first lidar coordinate system is the translation vector of the second lidar coordinate system relative to the first lidar coordinate system is the time synchronization error between the first lidar and the second lidar (unit: second).

[0039] Select the stable structures of planes and lines in the environment as the constraint plane and the constraint line respectively, and establish plane constraints and edge constraints

[0040] Plane constraint: For the constraint plane , its normal vector n should satisfy

[0041] where is the normal vector of the constraint plane, and the superscript T represents the transpose is the distance between the constraint plane of the m-th calibrated point and the origin of the first lidar coordinate system

[0042] Edge constraint: For the constraint line , the direction vector u satisfies

[0043] where is the direction vector of the constraint line in the second lidar point cloud data of the time-series points obtained from the second lidar scan is the direction vector of the constraint line in the first lidar point cloud data of the time-series points obtained from the first lidar scan

[0044] Combining the point distance error and the structure constraint, construct a differentiable optimization synchronization error function

[0045]

[0046]

[0047] Among them, is the synchronization error function, is the rotation matrix of the second lidar coordinate system relative to the first lidar coordinate system, is the translation vector of the second lidar coordinate system relative to the first lidar coordinate system, is the time synchronization error between the first lidar and the second lidar, is the point cloud registration term error function, is the plane constraint term error function; is the sequential point obtained by the second lidar scanning The three-dimensional point cloud of the m-th calibrated point in the point cloud data of the second lidar, is the sequential point obtained by the first lidar scanning The three-dimensional point cloud of the m-th calibrated point in the point cloud data of the first lidar, is the laser linear velocity, is the weight of the three-dimensional point cloud of the m-th calibrated point; is the plane constraint term error weight, is the normal vector of the constraint plane, and the superscript T represents the transpose, is the distance between the constraint plane of the m-th calibrated point and the origin of the first lidar coordinate system.

[0048] In another exemplary embodiment, the Lie algebra parameterization method can be used to solve the synchronization error function, but it is not limited to this solution method.

[0049] In another exemplary embodiment, in step 203 above, the original point cloud of the first lidar (i.e., the point cloud data of the first lidar) is converted into image forms such as a top-down bird's-eye view and a front-view depth map based on the point cloud multi-view method, providing necessary high-precision environmental perception information input for subsequent moving target state prediction.

[0050] In the embodiment of the present application, in order to convert the original point cloud of the vehicle-mounted lidar into a top-down bird's-eye view and a front-view depth map, it is obtained by using the Figure 4 multi-view generation process shown.

[0051] From the three-dimensional lidar original point cloud through the process of three-dimensional to two-dimensional projection, the bird's-eye view and depth map can be obtained by performing transformation according to the internal parameters of the corresponding camera. The conversion relationship from the lidar coordinate system to the image coordinate system is:

[0052] In the formula, is the pixel coordinate ， is the lidar coordinate, represents the rotation matrix from the lidar coordinate system to the camera coordinate system, represents the three-dimensional translation vector from the lidar coordinate system to the world coordinate system, 、 are the internal parameters of the camera, is the resolution of the image, is the camera focal length. is the pixel coordinate corresponding depth value.

[0053] The camera involved in the conversion process is a virtual camera, that is, a virtual camera is set to take pictures in the top view direction and the front view direction to obtain a bird's-eye view and a depth map.

[0054] In another exemplary embodiment, the target vehicle state prediction in a complex traffic scenario estimates the future state according to its own running state and in combination with the spatio-temporal interaction relationship of the surrounding environment. To effectively mine the interaction features in complex traffic data, the input data sensed by the second lidar below should include vehicle size parameters, running state parameters, etc.

[0055] The multi-target information interaction scenario in a complex traffic scenario is as Figure 5 shown. The intelligent vehicle equipped with a lidar sensor is located in the center of the road, and its running state model is: (3) In the formula, is the running state vector of the intelligent vehicle, L is the length of the intelligent vehicle; W is the width of the intelligent vehicle; 、 are the horizontal and vertical axis coordinate values of the center point of the second lidar at time t; is the instantaneous speed of the intelligent vehicle at time t; is the historical state data of the intelligent vehicle running, 、 are the horizontal and vertical axis coordinate values of the center point of the second lidar at time t - 5, is the instantaneous speed of the intelligent vehicle at time t - 5, 、 are the horizontal and vertical axis coordinate values of the center point of the second lidar at time t - 1, is the instantaneous speed of the intelligent vehicle at time t - 1.

[0056] The vehicle directly in front sensed by the second lidar of the intelligent vehicle 、 The vehicle in the right front 、 The vehicle in the left front The vehicle in the right rear are participating vehicles whose state behaviors need to be predicted. Taking the vehicle in the dead ahead as an example , its operation state model is as follows: (4) In the formula is the operation state vector of the vehicle in the dead ahead , is the length of the vehicle in the dead ahead ; is the width of the vehicle in the dead ahead ; , are the horizontal and vertical axis coordinate values of the center point of the vehicle in the dead ahead at time t; is the instantaneous speed of the vehicle in the dead ahead at time t; is the historical state data of the vehicle in the dead ahead , , are the horizontal and vertical axis coordinate values of the center point of the vehicle in the dead ahead at time t - 5, is the instantaneous speed of the vehicle in the dead ahead at time t - 5, , are the horizontal and vertical axis coordinate values of the center point of the vehicle in the dead ahead at time t - 1, is the instantaneous speed of the vehicle in the dead ahead at time t - 1.

[0057] The operation state data of the complex traffic scenario composed of the intelligent vehicle and the vehicle in the dead ahead , the vehicle in the right front , the vehicle in the left front and the vehicle in the right rear is: In the formula (5) In the formula is the operation state vector of the intelligent vehicle, are the operation state vectors of the 4 participating vehicles respectively, that is, the operation state vectors of the vehicle in the dead ahead , the vehicle in the right front , the vehicle in the left front and the vehicle in the right rear .

[0058] The overall network architecture of the multi-view feature extraction network, multi-object interaction information extraction network, and BiLSTM prediction network proposed in the embodiments of this application is as follows Figure 2 shown. By inputting the operating state data of multiple targets in a complex traffic scene obtained by sensors such as lidar and the corresponding bird's-eye view and depth map of the time series, the prediction network outputs the operating state data of multiple targets at future moments, thereby realizing the prediction of the risk field at future moments.

[0059] In another exemplary embodiment, the multi-view feature extraction network uses two improved VGG19 (Visual Geometry Group 19) networks to extract multi-view features of point clouds. One branch extracts depth map features, and the other branch extracts bird's-eye view features. The improved VGG19 network is based on a shallow CNN (Convolutional Neural Networks), and several convolutional layers are added. Since adding convolutional layers is more conducive to image feature extraction than adding fully connected layers, the improved VGG19 network is more likely to overcome the deficiencies of traffic scene diversity and complexity than the shallow convolutional neural network, and finally achieves better spatio-temporal feature extraction effects. As Figure 6 shown, the input layer image size of the multi-view feature extraction network provided in the embodiments of this application is 224×224 pixels, and the number of channels is 3; both VGG19 networks have 16 convolutional layers. Among them, max pooling layers are added after the 2nd, 4th, 8th, 12th, and 16th convolutional layers. The size of the convolutional kernels in the convolutional layers is gradually reduced by half from 224×224 to 14×14. Using gradually decreasing convolutional kernels is equivalent to adding implicit regularization, which can improve the network's feature extraction ability and increase the network's operation speed. The number of neurons in the 4 fully connected layers of the two VGG19s is 4096, 4096, 1000, and 5 respectively. The depth map features and bird's-eye view features corresponding to the time series points are concatenated in the last layer, and the high-level abstract combined features in the multi-view scenario are extracted from them to obtain multi-view features.

[0060] In another exemplary embodiment, the multi-object interaction relationship corresponds to a one-dimensional time series. In the embodiments of this application, a one-dimensional CNN network is used to extract potential interaction relationships. Specifically, local information is calculated by sliding a convolutional kernel of a specific size over the local area of the input data in turn.

[0061] As Figure 7As shown in the figure, the input of the multi-object interaction information extraction network is the motion state data of each target time series point in the complex traffic scene obtained based on the second lidar, and the output is the potential multi-object interaction features corresponding to the time series points extracted by the one-dimensional CNN network. The representation method of this multi-object interaction feature is as follows: each dynamic target (intelligent vehicle or participating vehicle) is represented by a node, and the nodes corresponding to any two dynamic targets in the same point cloud frame are connected by a solid line to represent the spatial edge, and the same target in adjacent frames is connected by a dotted line to represent the time edge.

[0062] In another exemplary embodiment, the output features of the multi-view feature extraction network and the multi-object interaction information extraction network are fused and input into the established BiLSTM prediction network for running state prediction. As Figure 8 shown, the BiLSTM prediction network consists of a BiLSTM layer, an Attention layer, an output layer, etc.

[0063] In another exemplary embodiment, in order to quickly quantify the driving risk level of a vehicle in the environment, the present application embodiment establishes a running risk field for each participating vehicle. This running risk field comprehensively considers the external dimensions of the participating vehicle and the relative motion between the participating vehicle and the intelligent vehicle. The running risk field is the sum of the static risk field and the dynamic risk field. The calculation formula of the running risk field is: (6) (7) (8) (9) (10) Among them, is the running risk field of the intelligent vehicle with respect to the jth participating vehicle at time is the static risk field of the intelligent vehicle with respect to the jth participating vehicle at time is the dynamic risk field of the intelligent vehicle with respect to the jth participating vehicle at time; is the field strength coefficient, B and C are the first intermediate function and the second intermediate function respectively, β is the high-order coefficient, and are the external shape parameters of the jth participating vehicle at time and are the lateral dimension coefficient and the longitudinal dimension coefficient of the jth participating vehicle at time and respectively the length and width of the j-th participating vehicle at time is the speed relationship parameter between the j-th parameter vehicle and the intelligent vehicle at time is the direction relationship parameter between the j-th parameter vehicle and the intelligent vehicle at time and respectively the abscissa and ordinate of the intelligent vehicle on the road surface at time, the direction of the abscissa is parallel to the road, and the ordinate is perpendicular to the road is the relative speed coefficient at time is the speed coefficient at time is the speed of the j-th participating vehicle at time is the speed of the intelligent vehicle at time

[0064] is the heading angle of the j-th participating vehicle on the road surface at time and respectively the abscissa and ordinate of the j-th participating vehicle on the road surface at time

[0065] According to the specific embodiments provided by the present application, the present application has the following technical effects

[0066] High prediction accuracy: By fusing the multi-view features of point cloud time series and multi-object interaction information, the motion state of the target vehicle can be predicted more accurately. Compared with traditional models such as FC, LSTM, and BiLSTM, the prediction effect is significantly improved

[0067] Comprehensive risk assessment: Based on the static and dynamic risk field models established by the artificial potential field theory, factors such as the shape and size of obstacles and relative motion are comprehensively considered, and the operation risks of intelligent vehicles can be comprehensively and real-time evaluated, providing a more reliable basis for autonomous driving decision-making

[0068] Strong adaptability: Applicable to complex traffic scenarios, it can effectively handle situations such as multi-vehicle interaction and different road conditions, and has strong environmental adaptability

[0069] Improved safety: By predicting and evaluating operation risks in advance, it provides support for decision-making such as obstacle avoidance and path planning of intelligent vehicles, helps reduce the incidence of traffic accidents, and improves the safety of autonomous driving

[0070] Based on the same inventive concept, an embodiment of the present application further provides an intelligent vehicle operation risk field prediction device for implementing the intelligent vehicle operation risk field prediction method involved above. The implementation solutions provided by this device to solve problems are similar to those recorded in the above method. Therefore, the specific limitations in one or more embodiments of the intelligent vehicle operation risk field prediction device provided below can refer to the limitations on the intelligent vehicle operation risk field prediction method in the above text, and will not be elaborated here.

[0071] In an exemplary embodiment, an intelligent vehicle operation risk field prediction device is provided, including: A fusion data acquisition module, configured to synchronously acquire bird's-eye view maps, depth maps, and operation state data of each target time point by using a first lidar and a second lidar installed on the intelligent vehicle; each target time point includes the current time point and a preset number of historical time points before the current time point; the first lidar is used to acquire point cloud data representing the environmental global map information of the intelligent vehicle, and the second lidar is used to acquire point cloud data representing the operation state information of the intelligent vehicle and participating vehicles; the participating vehicles include vehicles within the scanning range of the second lidar of the intelligent vehicle.

[0072] A multi-view feature extraction module, configured to extract features from the bird's-eye view maps and depth maps of each target time point by using a multi-view feature extraction network to obtain multi-view features of each target time point.

[0073] A multi-target interaction feature extraction module, configured to extract multi-target interaction features from the motion state data of each target time point by using a multi-target interaction information extraction network to obtain multi-target interaction features of each target time point.

[0074] A splicing module, configured to splice the multi-view features and multi-target interaction features of each target time point respectively to obtain the fusion features of each target time point.

[0075] A prediction module, configured to obtain the operation state data of the predicted time point based on the fusion features of each target time point by using a BiLSTM prediction network; the predicted time point is after the current time point; A risk field calculation module, configured to calculate the operation risk field of the intelligent vehicle based on the operation state data of the predicted time point.

[0076] In another exemplary embodiment, the fusion data acquisition module includes: a dual multi-line lidar environmental perception module, a point cloud multi-view generation module, and a multi-target information interaction network module.

[0077] The dual multi-line lidar environmental perception module: configured to acquire the interaction information between the surrounding environment of the intelligent vehicle and traffic participants.

[0078] Point cloud multi-view generation module: used to convert the original point cloud data into a bird's-eye view and a depth map.

[0079] Multi-object information interaction network module: used to extract the motion state data of multiple objects in a complex traffic scene.

[0080] In another exemplary embodiment, the above multi-view feature extraction module, multi-object interaction feature extraction module, stitching module, prediction module, and risk field calculation module constitute a vehicle operation risk prediction module. This vehicle operation risk prediction module: includes a multi-view feature extraction network, a multi-object interaction information extraction network, a BiLSTM prediction network, and an operation risk assessment network, and is used to predict the motion state of the target vehicle and evaluate the operation risk of the intelligent vehicle.

[0081] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0082] Specific examples are used in this article to elaborate on the principles and implementation manners of this application. The descriptions of the above embodiments are only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. An intelligent vehicle operation risk field prediction method, characterized in that Including: Utilize the first lidar and the second lidar installed on the intelligent vehicle to synchronously obtain the bird's-eye view, depth map, and operating status data of each target time series point; each target time series point includes the current time series point and a preset number of historical time series points before the current time series point; the first lidar is used to obtain point cloud data representing the environmental global map information of the intelligent vehicle, and the second lidar is used to obtain point cloud data representing the operating status information of the intelligent vehicle and the participating vehicles; the participating vehicles include the vehicles within the scanning range of the second lidar of the intelligent vehicle; Adopt a multi-view feature extraction network to extract features from the bird's-eye view and depth map of each target time series point, and obtain the multi-view features of each target time series point; Adopt a multi-object interaction information extraction network to perform multi-object interaction feature extraction on the motion state data of each target time series point, and obtain the multi-object interaction features of each target time series point; Concatenate the multi-view features and multi-object interaction features of each target time series point respectively to obtain the fusion features of each target time series point; According to the fusion features of each target time series point, use the BiLSTM prediction network to obtain the operating status data of the predicted time series point; the predicted time series point is after the current time series point; Based on the operating status data of the predicted time series point, calculate the operating risk field of the intelligent vehicle.

2. The intelligent vehicle operation risk field prediction method according to claim 1, wherein Both the first lidar and the second lidar are installed on the top of the intelligent vehicle, the first lidar is located above the second lidar, and the installation height and installation angle of the first lidar and the second lidar satisfy the following formula: ； Among them, is the maximum projection distance of the scan line of the first lidar on the ground, is the minimum projection distance of the scan line of the first lidar on the ground, is the installation height of the first radar, is the angle between the horizontal line and the lowermost scan line of the first lidar, representing the installation angle, is the vertical angular resolution of the first lidar, is the i-th scan line of the first lidar, representing the corresponding scan line of the first lidar; is the maximum projection distance of the scan line of the second lidar on the ground, is the minimum projection distance of the scan line of the second lidar on the ground, is the installation height of the second radar, is the angle between the horizontal line and the lowermost scan line of the second lidar, representing the installation angle, is the vertical angular resolution of the second lidar, is the i-th scan line of the second lidar, representing the corresponding scan line of the second lidar.

3. The intelligent vehicle operation risk field prediction method according to claim 1, characterized in that Utilize the first lidar and the second lidar installed on the intelligent vehicle to synchronously obtain the bird's-eye view, depth map, and operating status data, specifically including: Solve the time synchronization error between the first lidar and the second lidar; According to the time synchronization error, control the first lidar and the second lidar respectively to synchronously obtain the first lidar point cloud data and the second lidar point cloud data of each target time series point; Convert the first lidar point cloud data of each target time series point into a bird's-eye view and a depth map; Extract the motion features from the second lidar point cloud data of each target time series point to obtain the operating status data of each target time series point; the operating status data includes: the operating status vector of the intelligent vehicle and the operating status vectors of each participating vehicle.

4. The intelligent vehicle operation risk field prediction method according to claim 3, wherein, Solve the time synchronization error between the first lidar and the second lidar, specifically including: Construct a spatio-temporal joint compensation model and a structural feature constraint for the first lidar and the second lidar; According to the spatio-temporal joint compensation model and the structural feature constraint, construct a synchronization error function; Optimize and solve the synchronization error function to obtain the time synchronization error.

5. The intelligent vehicle operation risk field prediction method according to claim 4, wherein, The spatio-temporal joint compensation model is: ； Among them, is the three-dimensional point cloud of the m-th calibration point in the second lidar point cloud data obtained by the second lidar scan ; is the three-dimensional point cloud of the m-th calibration point in the first lidar point cloud data obtained by the first lidar scan ; is the linear velocity of the laser; is the rotation matrix of the second lidar coordinate system relative to the first lidar coordinate system; is the translation vector of the second lidar coordinate system relative to the first lidar coordinate system; is the time synchronization error between the first lidar and the second lidar.

6. The intelligent vehicle operation risk field prediction method according to claim 4, characterized in that, The structural feature constraints include plane constraints and edge constraints; The plane constraint is: ； Among them, is the time-series point obtained by the second lidar scan and is the three-dimensional point cloud of the m-th calibration point in the second lidar point cloud data, is the linear velocity of the laser, is the rotation matrix of the second lidar coordinate system relative to the first lidar coordinate system, is the translation vector of the second lidar coordinate system relative to the first lidar coordinate system, is the time synchronization error between the first lidar and the second lidar, is the normal vector of the constraint plane, and the superscript T represents the transpose, is the distance between the constraint plane of the m-th calibration point and the origin of the first lidar coordinate system; The edge constraint is: ； Among them, is the direction vector of the constraint line in the second lidar point cloud data of the time series points obtained by the second lidar scan, and is the direction vector of the constraint line in the first lidar point cloud data of the time series points obtained by the first lidar scan. ​ 7. The intelligent vehicle operation risk field prediction method according to claim 4, wherein The synchronization error function is: ； ； ； wherein, is the synchronization error function, is the rotation matrix of the second lidar coordinate system relative to the first lidar coordinate system, is the translation vector of the second lidar coordinate system relative to the first lidar coordinate system, is the time synchronization error between the first lidar and the second lidar, is the point cloud registration term error function, is the plane constraint term error function; is the time series point obtained by the second lidar scan The 3D point cloud of the m-th calibration point in the second lidar point cloud data, is the time series point obtained by the first lidar scan The 3D point cloud of the m-th calibration point in the first lidar point cloud data, is the laser linear velocity, is the weight of the 3D point cloud of the m-th calibration point; is the plane constraint term error weight, is the normal vector of the constraint plane, and the superscript T represents the transpose, is the distance between the constraint plane of the m-th calibration point and the origin of the first lidar coordinate system.

8. The intelligent vehicle operation risk field prediction method according to claim 1, wherein The multi-view feature extraction network includes: the first VGG19 network, the second VGG19 network, and a concatenation network; The first VGG19 network is used to extract features from the bird's-eye view image to obtain bird's-eye view features; The second VGG19 network is used to extract features from the depth map to obtain depth map features; The splicing network is used to connect the bird's-eye view features and the depth map features to obtain multi-view features.

9. The intelligent vehicle operation risk field prediction method according to claim 8, wherein Both the first VGG19 network and the second VGG19 network are obtained by adding convolutional layers to the CNN network; Both the first VGG19 network and the second VGG19 network include an input layer, 16 convolutional layers, 5 max-pooling layers, a flattening layer, and 4 fully-connected layers; The 16 convolutional layers are connected in sequence, and the 5 pooling layers are sequentially arranged after the 2nd, 4th, 8th, 12th, and 16th convolutional layers, and the input layer is arranged before the 1st convolutional layer; The max-pooling layer after the 16th convolutional layer is connected to the flattening layer, and 4 fully-connected layers are sequentially connected after the flattening layer. Among them, the last fully-connected layer serves as the output layer, and the output layer is connected to the splicing network.

10. An intelligent vehicle operation risk field prediction device, characterized in that, The intelligent vehicle operation risk field prediction device applies the intelligent vehicle operation risk field prediction method according to any one of claims 1-9. The intelligent vehicle operation risk field prediction device includes: A fusion data acquisition module, which is used to synchronously acquire the bird's-eye view image, depth map, and operation state data of each target time series point by using the first lidar and the second lidar installed on the intelligent vehicle; each target time series point includes the current time series point and a preset number of historical time series points before the current time series point; the first lidar is used to acquire point cloud data representing the environmental global map information of the intelligent vehicle, and the second lidar is used to acquire point cloud data representing the operation state information of the intelligent vehicle and the participating vehicles; the participating vehicles include the vehicles within the scanning range of the second lidar of the intelligent vehicle; A multi-view feature extraction module, which is used to extract features from the bird's-eye view image and depth map of each target time series point by using a multi-view feature extraction network to obtain multi-view features of each target time series point; A multi-target interaction feature extraction module, which is used to extract multi-target interaction features from the motion state data of each target time series point by using a multi-target interaction information extraction network to obtain multi-target interaction features of each target time series point; A splicing module, which is used to splice the multi-view features and multi-target interaction features of each target time series point respectively to obtain the fusion features of each target time series point; A prediction module, which is used to obtain the operation state data of the predicted time series point by using a BiLSTM prediction network according to the fusion features of each target time series point; the predicted time series point is located after the current time series point; A risk field calculation module, which is used to calculate the operation risk field of the intelligent vehicle based on the operation state data of the predicted time series point.

Images