Obstacle behavior prediction method based on feature interaction and autonomous vehicle

Abstract

The present disclosure provides a feature interaction-based obstacle behavior prediction method and an autonomous vehicle, relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, image processing and computer vision, and can be applied to the field of autonomous driving. A specific embodiment of the method comprises: acquiring obstacle and driving environment information around a vehicle; extracting obstacle features and driving environment features from the obstacle and driving environment information; performing feature interaction on the obstacle features and the driving environment features to obtain interaction features; inputting the interaction features into a decoder for motion prediction to obtain a future behavior state of the obstacles around the vehicle. This embodiment enhances the expression capability of interaction, thereby improving the prediction accuracy.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, image processing and computer vision, and can be applied to the field of automatic driving. BACKGROUND

[0002] Accurate prediction of the future behavior state of an obstacle is crucial for the safe, comfortable and efficient operation of an automatic driving vehicle. Predicting the future behavior state requires understanding multiple static and dynamic information around the automatic driving vehicle, including but not limited to road topology, dynamic traffic signal information, and historical motion information of the obstacle, etc.

[0003] Currently, the obstacle behavior prediction method usually uses a deep learning model to process multiple static and dynamic information around the automatic driving vehicle respectively, and then predicts the future behavior state of the obstacle based on the respective processing results. SUMMARY

[0004] Embodiments of the present disclosure provide an obstacle behavior prediction method and device based on feature interaction, an equipment, a storage medium, a program product and an automatic driving vehicle.

[0005] In a first aspect, the embodiments of the present disclosure provide an obstacle behavior prediction method based on feature interaction, comprising: obtaining obstacle and driving environment information around a vehicle; extracting obstacle features and driving environment features from the obstacle and driving environment information; performing feature interaction on the obstacle features and the driving environment features to obtain interaction features; inputting the interaction features into a decoder for motion prediction to obtain a future behavior state of an obstacle around the vehicle; wherein the obstacle and driving environment information comprises surrounding perception information and surrounding map information of the vehicle, the driving environment features comprise stop line features and road features; and the extracting of the obstacle features and the driving environment features from the obstacle and driving environment information comprises: extracting the obstacle features and the stop line features from the surrounding perception information, and extracting the road features from the surrounding map information.

[0006] In a second aspect, the embodiments of the present disclosure provide a device for predicting behavior of an obstacle based on feature interaction, comprising: an obtaining module configured to obtain an obstacle and driving environment information around a vehicle; an extracting module configured to extract an obstacle feature and a driving environment feature from the obstacle and driving environment information; an interaction module configured to perform feature interaction on the obstacle feature and the driving environment feature to obtain an interaction feature; and a predicting module configured to input the interaction feature into a decoder to perform motion prediction to obtain a future behavior state of the obstacle around the vehicle; wherein the obstacle and driving environment information comprises surrounding perception information and surrounding map information of the vehicle, the driving environment feature comprises a stop line feature and a road feature; and the extracting module is further configured to extract the obstacle feature and the stop line feature from the surrounding perception information, and extract the road feature from the surrounding map information.

[0007] In a third aspect, the embodiments of the present disclosure provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any one of the implementations of the first aspect.

[0008] In a fourth aspect, the embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause a computer to perform the method described in any one of the implementations of the first aspect.

[0009] In a fifth aspect, the embodiments of the present disclosure provide a computer program product, comprising a computer program, the computer program being executed by a processor to implement the method described in any one of the implementations of the first aspect.

[0010] In a sixth aspect, the embodiments of the present disclosure provide an autonomous vehicle, comprising the electronic device described in any one of the implementations of the third aspect.

[0011] The method for predicting behavior of an obstacle based on feature interaction provided by the embodiments of the present disclosure first performs feature interaction on an obstacle feature and a driving environment feature, and then performs motion prediction based on an interaction feature. Through feature interaction, the interaction feature contains the correlation between the obstacle and the driving environment, the expression ability of the interaction is enhanced, and the prediction accuracy is improved.

[0012] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description when read in conjunction with the accompanying drawings. The drawings are provided for the purpose of illustration only and are not intended to limit the scope of the present disclosure. In the drawings:

[0014] Figure 1 is a flowchart of one embodiment of a feature interaction based obstacle behavior prediction method according to the present disclosure;

[0015] Figure 2 is a flowchart of another embodiment of a feature interaction based obstacle behavior prediction method according to the present disclosure;

[0016] Figure 3 is a flowchart of another embodiment of a feature interaction based obstacle behavior prediction method according to the present disclosure;

[0017] Figure 4 is a flowchart of another embodiment of a feature interaction based obstacle behavior prediction method according to the present disclosure;

[0018] Figure 5 is a network structure diagram of feature embedding;

[0019] Figure 6 is a network structure diagram of multi-level feature interaction;

[0020] Figure 7 is a structural schematic diagram of one embodiment of a feature interaction based obstacle behavior prediction apparatus according to the present disclosure;

[0021] Figure 8 is a block diagram of an electronic device for implementing a feature interaction based obstacle behavior prediction method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help the understanding of the present disclosure. These should be considered in the context of the overall description and should not be considered limiting in any way. Thus, it will be apparent to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted from the following description.

[0023] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0024] Figure 1A flow 100 of one embodiment of a feature interaction based obstacle behavior prediction method according to the present disclosure is shown. The feature interaction based obstacle behavior prediction method includes the following steps:

[0025] Step 101, obtaining obstacle and driving environment information of a vehicle periphery.

[0026] In this embodiment, the execution subject of the feature interaction based obstacle behavior prediction method can obtain obstacle and driving environment information of a vehicle periphery.

[0027] Generally, an automatic driving vehicle is installed with vehicle-mounted sensors. During vehicle driving, the vehicle periphery is perceived by the vehicle-mounted sensors to obtain obstacle and driving environment information. The vehicle-mounted sensors can include, but are not limited to, cameras, millimeter wave radars, laser radars, etc. The obstacle and driving environment information can include, but are not limited to, image data, video data, point cloud data, etc. The obstacle generally refers to dynamic obstacles in the vehicle periphery, such as motor vehicles, non-motor vehicles, pedestrians, and other road users. The driving environment generally includes road topology, traffic signal lights, stop lines, etc.

[0028] Step 102, extracting obstacle features and driving environment features from the obstacle and driving environment information.

[0029] In this embodiment, the above execution subject can extract obstacle features and driving environment features from the obstacle and driving environment information.

[0030] Generally, the obstacle and driving environment information is input into a CNN (Convolutional Neural Networks) to extract obstacle features and driving environment features. The convolution kernel of the CNN is the core of the CNN, which enables the CNN to construct information features by fusing spatial and channel information in the local receptive field of each layer. The convolutional neural network is composed of a series of convolutional layers, nonlinear layers and down-sampling layers, which can capture the features of the image from the theoretical global receptive field to describe the image. The obstacle features can express obstacle information. The driving environment features can express driving environment information. The obstacle features and the driving environment features are usually presented in the form of feature maps.

[0031] Step 103, performing feature interaction on the obstacle features and the driving environment features to obtain interaction features.

[0032] In this embodiment, the above execution subject can perform feature interaction on the obstacle features and the driving environment features to obtain interaction features.

[0033] Generally, the obstacle features and the driving environment features are interacted to make the interaction features contain the correlation between the obstacle and the driving environment, and enhance the expression ability of the interaction.

[0034] In step 104, the interaction features are input into the decoder for motion prediction to obtain the future behavior state of the obstacle around the vehicle.

[0035] In the embodiment, the execution subject can input the interaction features into the decoder for motion prediction to obtain the future behavior state of the obstacle around the vehicle.

[0036] Generally, the interaction features are motion predicted by using the decoder as a prediction head to obtain the future behavior state of the obstacle. The interaction features can be adapted to various decoders, which are not limited here.

[0037] The feature interaction-based obstacle behavior prediction method provided in the embodiments of the present disclosure first interacts the obstacle features with the driving environment features, and then performs motion prediction based on the interaction features. Through the feature interaction, the interaction features contain the correlation between the obstacle and the driving environment, and the expression ability of the interaction is enhanced, thereby improving the prediction accuracy.

[0038] With reference to Figure 2 Fig. 200 shows another embodiment of the feature interaction-based obstacle behavior prediction method according to the present disclosure. The feature interaction-based obstacle behavior prediction method comprises the following steps:

[0039] In step 201, the surrounding perception information and the surrounding map information of the vehicle are obtained.

[0040] In the embodiment, the execution subject of the feature interaction-based obstacle behavior prediction method can obtain the surrounding perception information and the surrounding map information of the vehicle. The obstacle and the driving environment information can include the surrounding perception information and the surrounding map information of the vehicle.

[0041] Generally, the automatic driving vehicle is installed with a vehicle-mounted sensor and a GPS (Global Positioning System). During the driving of the vehicle, the vehicle-mounted sensor is used to perceive the surroundings of the vehicle to obtain the surrounding perception information, and the GPS is used to locate the vehicle to obtain the surrounding map information. The vehicle-mounted sensor can include but is not limited to a camera, a millimeter wave radar, a laser radar, etc. The surrounding perception information can include but is not limited to image data, video data, point cloud data, etc. The surrounding map information is generally a high-definition map.

[0042] In step 202, the obstacle features and the stop line features are extracted from the surrounding perception information, and the road features are extracted from the surrounding map information.

[0043] In the embodiment, the execution subject can extract the obstacle feature and the stop line feature from the surrounding perception information, and extract the road feature from the surrounding map information. The driving environment feature can include the stop line feature and the road feature.

[0044] Generally, the surrounding perception information is input into the CNN, and the obstacle feature and the stop line feature can be extracted. The surrounding map information is input into the CNN, and the road feature can be extracted. The obstacle feature can express the obstacle information. The road feature can express the road topology information. The stop line feature can express the stop line and traffic light information. The obstacle feature, the road feature and the stop line feature can be serialized features.

[0045] Taking the obstacle feature with a size of 16 as an example, the detailed information can be shown in the following table:

[0046]

[0047] “end_point.x” is the horizontal coordinate of the end point, “end_point.y” is the vertical coordinate of the end point, “=5” is a placeholder field, “dis_to_adc” is the distance from the autonomous vehicle, “heading” is the direction, “length” is the length, “width” is the width, “type” is the type, “sub_type” is the sub-type, “velocity_converged” is the convergence velocity, “tracking_time” is the tracking time, “adc_theta” is the direction of the head of the autonomous vehicle, “is_interpolayion_vec” is whether linear interpolation is performed, “speed” is the speed, “linear_acc” is the acceleration, and “turn_signal” is the turn signal.

[0048] Taking the road feature with a size of 16 as an example, the detailed information can be shown in the following table:

[0049]

[0050] "end_point.x" is the horizontal coordinate of the end point, "end_point.y" is the vertical coordinate of the end point, "start_point.x" is the horizontal coordinate of the start point, "start_point.y" is the vertical coordinate of the start point, "0" is a placeholder, "lane_type" is a road type, "lane_turn_type" is a road turning type, "merge_fork_type" is a merging and branching type, "main_side_type" is a main side type, "speed_limit" is a speed limit, "heading" is a heading, "width" is a width, "lane_num" is a road number, "is_virtual" is whether it is a virtual road, "road_form" is a road type, and "signal_state" is a signal state.

[0051] For example, the detailed information of the stop line feature with a size of 16 can be as shown in the following table:

[0052] 0 end_point.x 4 4 8 left_turn_status 12 — 1 end_point.y 5 color 9 right_turn_status 13 — 2 start_point.x 6 source 10 uturn_status 14 — 3 start_point.y 7 noturn_status 11 — 15 —

[0053] "end_point.x" is the horizontal coordinate of the end point, "end_point.y" is the vertical coordinate of the end point, "start_point.x" is the horizontal coordinate of the start point, "start_point.y" is the vertical coordinate of the start point, "4" is a placeholder, "color" is a color, "source" is a source, "noturn_status" is a no-turn status, "left_turn_status" is a left-turn status, "right_turn_status" is a right-turn status, and "uturn_status" is a U-turn status.

[0054] In step 203, the obstacle feature and the driving environment feature are input to an encoder for multi-level feature interaction to obtain an interaction feature.

[0055] In the embodiment, the execution subject can input the obstacle feature and the driving environment feature to an encoder for multi-level feature interaction to obtain an interaction feature.

[0056] Generally, the obstacle feature and the driving environment feature are interacted by the encoder for multi-level feature interaction, and through multiple interactions of the obstacle feature and the driving environment feature, the association between the obstacle and the driving environment in the interaction feature is more closely, and the expression ability of the interaction is stronger.

[0057] In step 204, the interaction feature is input to a decoder for motion prediction to obtain a future behavior state of the obstacle around the vehicle.

[0058] In the embodiment, the execution subject can input the interaction feature to the decoder for motion prediction to obtain the future behavior state of the obstacle in the vehicle surroundings.

[0059] Generally, the motion prediction of the interaction feature by using the decoder as the prediction head can obtain the future behavior state of the obstacle. The interaction feature can be adapted to various decoders, which are not limited here.

[0060] As can be seen from Figure 2 , compared with the embodiment corresponding to Figure 1 , the process 200 of the obstacle behavior prediction method based on feature interaction in the embodiment highlights the feature interaction step. Therefore, the scheme described in the embodiment utilizes the encoder to perform multi-level feature interaction between the obstacle feature and the driving environment feature, and through multiple interactions between the obstacle feature and the driving environment feature, the correlation between the obstacle and the driving environment in the interaction feature is closer, the expression ability of the interaction is stronger, and the prediction accuracy is higher.

[0061] Further reference is made to Figure 3 , which shows a process 300 of another embodiment of the obstacle behavior prediction method based on feature interaction according to the present disclosure. The obstacle behavior prediction method based on feature interaction comprises the following steps:

[0062] Step 301, obtaining the surrounding perception information and the surrounding map information of the vehicle.

[0063] In the embodiment, the execution subject of the obstacle behavior prediction method based on feature interaction can obtain the surrounding perception information and the surrounding map information of the vehicle. The obstacle and driving environment information can include the surrounding perception information and the surrounding map information of the vehicle.

[0064] Generally, the autonomous vehicle is equipped with vehicle-mounted sensors and GPS. During the driving of the vehicle, the vehicle-mounted sensors are used to perceive the surroundings of the vehicle to obtain the surrounding perception information, and the GPS is used to locate the vehicle to further obtain the surrounding map information. The vehicle-mounted sensors can include but are not limited to cameras, millimeter wave radars, laser radars, etc. The surrounding perception information can include but is not limited to image data, video data, point cloud data, etc. The surrounding map information is generally a high-definition map.

[0065] Step 302, extracting the obstacle feature and the stop line feature from the surrounding perception information, and extracting the road feature from the surrounding map information.

[0066] In the embodiment, the execution subject can extract the obstacle feature and the stop line feature from the surrounding perception information, and extract the road feature from the surrounding map information. The driving environment feature can include the stop line feature and the road feature.

[0067] Generally, the peripheral awareness information is input into the CNN, and the obstacle feature and the stop line feature can be extracted. The peripheral map information is input into the CNN, and the road feature can be extracted. The obstacle feature can represent the obstacle information. The road feature can represent the road topology information. The stop line feature can represent the stop line and traffic light information. The obstacle feature, the road feature and the stop line feature can be sequential features.

[0068] In step 303, the obstacle feature, the stop line feature and the road feature are embedded to obtain the obstacle embedded feature and the road embedded feature.

[0069] In the embodiment, the execution subject mentioned above can embed the obstacle feature, the stop line feature and the road feature to obtain the obstacle embedded feature and the road embedded feature. Generally, the feature embedding can be performed by using the CNN.

[0070] In step 304, the attention mechanism is used to perform multi-level feature interaction on the obstacle embedded feature and the road embedded feature to obtain the interaction feature.

[0071] In the embodiment, the execution subject mentioned above can use the attention mechanism to perform multi-level feature interaction on the obstacle embedded feature and the road embedded feature to obtain the interaction feature. Generally, the feature interaction can be performed by using the attention mechanism, and the interaction relationship can be repeatedly learned. For different features, the attention mechanism can be used for feature interaction; for the same feature, the self-attention mechanism can be used for feature interaction. After multi-level feature interaction, the association between the obstacle and the driving environment in the interaction feature is closer, and the expression ability of the interaction is stronger.

[0072] In step 305, the interaction feature is input into the decoder for motion prediction to obtain the future behavior state of the obstacle in the vehicle periphery.

[0073] In the embodiment, the execution subject mentioned above can input the interaction feature into the decoder for motion prediction to obtain the future behavior state of the obstacle in the vehicle periphery.

[0074] Generally, the interaction feature can be motion predicted by using the decoder as the prediction head to obtain the future behavior state of the obstacle. The interaction feature can be adapted to various decoders, which are not limited here.

[0075] As can be seen from Figure 3 , compared with Figure 2Compared with the corresponding embodiments, the process 300 of the obstacle behavior prediction method based on feature interaction in the present embodiment highlights the feature interaction step. Thus, the scheme described in the present embodiment firstly uses CNN for feature embedding, and then uses attention mechanism for multi-level feature interaction, repeatedly learns the interaction relationship, greatly enhances the association between the obstacle and the driving environment in the interactive features, greatly enhances the expression ability of the interaction, and further improves the prediction accuracy.

[0076] Further reference is made to Figure 4 which shows a process 400 of still another embodiment of the obstacle behavior prediction method based on feature interaction according to the present disclosure. The obstacle behavior prediction method based on feature interaction comprises the following steps:

[0077] Step 401, obtaining the surrounding perception information and the surrounding map information of the vehicle.

[0078] In the present embodiment, the execution subject of the obstacle behavior prediction method based on feature interaction can obtain the surrounding perception information and the surrounding map information of the vehicle. Among them, the obstacle and driving environment information can include the surrounding perception information and the surrounding map information of the vehicle.

[0079] Generally, an automatic driving vehicle is installed with an on-board sensor and a GPS. During the driving of the vehicle, the surrounding perception information is obtained by the on-board sensor sensing the surroundings of the vehicle, and the surrounding map information is obtained by the GPS positioning the vehicle. Among them, the on-board sensor can include but is not limited to a camera, a millimeter wave radar, a laser radar, etc. The surrounding perception information can include but is not limited to image data, video data, point cloud data, etc. The surrounding map information is usually a high-precision map.

[0080] Step 402, extracting the obstacle feature and the stop line feature from the surrounding perception information, and extracting the road feature from the surrounding map information.

[0081] In the present embodiment, the above execution subject can extract the obstacle feature and the stop line feature from the surrounding perception information, and extract the road feature from the surrounding map information. Among them, the driving environment feature can include the stop line feature and the road feature.

[0082] Generally, the surrounding perception information is input into the CNN, and the obstacle feature and the stop line feature can be extracted. The surrounding map information is input into the CNN, and the road feature can be extracted. The obstacle feature can express the obstacle information. The road feature can express the road topology information. The stop line feature can express the stop line and traffic light information. The obstacle feature, the road feature and the stop line feature can be serialized features.

[0083] Step 403, input the obstacle feature, the stop line feature and the road feature into the corresponding convolution layer respectively to obtain the obstacle intermediate feature, the stop line intermediate feature and the road intermediate feature.

[0084] In the embodiment, the execution subject can input the obstacle feature, the stop line feature and the road feature into the corresponding convolution layer respectively, reduce the input features of different dimensions to the same dimension to obtain the obstacle intermediate feature, the stop line intermediate feature and the road intermediate feature.

[0085] Generally, the number of convolution layers is proportional to the dimension of the input feature of the convolution layer. That is, the higher the dimension of the input feature of the convolution layer, the more the number of the corresponding convolution layer, so that the input features of different dimensions can be reduced to the same dimension. In practical application, the obstacle feature and the road feature contain more information and have larger dimension, and the stop line feature contains less information and has smaller dimension, therefore, the number of the convolution layer corresponding to the obstacle feature and the road feature is generally more than the number of the convolution layer corresponding to the stop line feature. For example, the obstacle feature and the road feature are input into a plurality of convolution layers connected in series respectively to obtain the obstacle intermediate feature and the road intermediate feature; the stop line feature is input into one convolution layer to obtain the stop line intermediate feature.

[0086] Step 404, splice the obstacle intermediate feature and the stop line intermediate feature to obtain the obstacle embedding feature, and take the road intermediate feature as the road embedding feature.

[0087] In the embodiment, the execution subject can splice the obstacle intermediate feature and the stop line intermediate feature to obtain the obstacle embedding feature, and take the road intermediate feature as the road embedding feature. That is, the obstacle intermediate feature and the stop line intermediate feature are aggregated, and the road intermediate feature is processed separately.

[0088] In order to facilitate understanding, Figure 5 The network structure diagram of the feature embedding is shown. As shown in Figure 5As shown, the obstacle feature obstacle feature[B, 32, 16, 16] and the road feature lane feature[B, 64, 64, 16] are respectively input into two serial convolutional layers Conv1D to obtain the obstacle intermediate feature[B, 32, D] and the road intermediate feature[B, 64, D]. The stop line feature stopline feature[B, 16, 8, 16] is input into a Conv1D to obtain the stop line intermediate feature[B, 16, D]. The obstacle intermediate feature[B, 32, D] and the stop line intermediate feature[B, 16, D] are spliced to obtain the obstacle embedding feature obstacle, and the road intermediate feature[B, 32, D] is taken as the road embedding feature lane. In the obstacle feature obstacle feature[B, 32, 16, 16], “B” is the number of samples selected each time for training, “32” is the number of obstacles, the first “16” is the time dimension, 16 frames, about 1.6s, and the second “16” is the feature dimension. In the lane feature lane feature[B, 64, 64, 16], “B” is the number of samples selected each time for training, the first “64” is the number of polyline, the second “64” is the number of line segments in each polyline, and “16” is the feature dimension. In the stop line feature stopline feature[B, 16, 8, 16], “B” is the number of samples selected each time for training, the first “16” is the time dimension, “8” is the number of polylines, and the second “16” is the feature dimension. In the intermediate feature, “B” is the number of samples selected each time for training, and “D” is the hidden layer feature dimension.

[0089] In step 405, the obstacle embedding feature and the road embedding feature are interacted by using an attention mechanism to obtain the first interaction feature of the vehicle to the road.

[0090] In the embodiment, the execution subject can interact the obstacle embedding feature and the road embedding feature by using an attention mechanism, introduce real-time traffic information into the lane node, for example, the congestion or usage of the lane, to obtain the first interaction feature of the vehicle to the road.

[0091] In step 406, the first interaction feature is interacted by using a self-attention mechanism to obtain the second interaction feature of the road to the road.

[0092] In the embodiment, the execution subject can interact the first interaction feature by using a self-attention mechanism, further perform interaction encoding, obtain the second interaction feature of the road to the road, and update the lane node feature by propagating traffic information on the lane graph.

[0093] Step 407, the second interaction feature and the obstacle embedding feature are interacted by using an attention mechanism to obtain a third interaction feature of the road to the vehicle.

[0094] In the embodiment, the execution subject can interact the second interaction feature and the obstacle embedding feature by using an attention mechanism to obtain a third interaction feature of the road to the vehicle, and fuse the updated map feature and the real-time traffic information back to the participants.

[0095] Step 408, the third interaction feature is interacted by using a self-attention mechanism to obtain a fourth interaction feature of the vehicle to the vehicle.

[0096] In the embodiment, the execution subject can interact the third interaction feature by using a self-attention mechanism to obtain a fourth interaction feature of the vehicle to the vehicle, process the interaction between the obstacles, and generate the final interaction feature of the obstacles and the environment.

[0097] In order to facilitate understanding, Figure 6 A network structure diagram of multi-level feature interaction is shown. As shown in Figure 6 The first step, the attention mechanism is used for the bottom-level interaction of the encoded obstacle embedding feature and the road embedding feature, the real-time traffic information is introduced into the lane node, for example, the congestion or usage of the lane, the interaction information of a2l (agent to lane, vehicle to road) is obtained; the second step, the self-attention mechanism is used for the a2l to further interact and encode to obtain the interaction information of l2l (lane to lane, road to road), the lane node feature is updated by propagating the traffic information on the lane graph; the third step, the l2l and the encoded obstacle embedding feature obstacle do attention to obtain the interaction information of l2a (lane to agent, road to vehicle), the updated map feature and the real-time traffic information are fused back to the participants; the fourth step, the l2a does self-attention to obtain the interaction information of a2a (agent to agent, vehicle to vehicle), processes the interaction between the obstacles, and generates the final interaction feature of the obstacles and the environment, and then the prediction head is used for motion prediction.

[0098] Step 409, the interaction feature is input to the decoder for motion prediction to obtain the future behavior state of the obstacles around the vehicle.

[0099] In the embodiment, the execution subject can input the interaction feature to the decoder for motion prediction to obtain the future behavior state of the obstacles around the vehicle.

[0100] Generally, the future behavior state of the obstacle can be obtained by using the decoder as a prediction head to perform motion prediction on the interaction feature. The interaction feature can be adapted to various decoders, which are not limited here.

[0101] As can be seen from Figure 4 , compared with the embodiment corresponding to Figure 3 , the flow 400 of the obstacle behavior prediction method based on feature interaction in the embodiment highlights the steps of feature embedding and feature interaction. Thus, the scheme described in the embodiment first uses a convolutional layer to perform feature embedding, and then uses an attention mechanism to perform multi-level feature interaction, repeatedly learns the interaction relationship, greatly enhances the correlation between the obstacle and the driving environment in the interaction feature, greatly enhances the expression ability of the interaction, and further improves the prediction accuracy.

[0102] Further referring to Figure 7 , as an implementation of the method shown in the above figures, the disclosure provides an embodiment of an obstacle behavior prediction device based on feature interaction. The device embodiment corresponds to the method embodiment shown in Figure 1 , and the device can be specifically applied to various electronic devices.

[0103] As shown in Figure 7 , the obstacle behavior prediction device 700 based on feature interaction in the embodiment can include an acquisition module 701, an extraction module 702, an interaction module 703, and a prediction module 704. The acquisition module 701 is configured to acquire obstacle and driving environment information around a vehicle. The extraction module 702 is configured to extract obstacle features and driving environment features from the obstacle and driving environment information. The interaction module 703 is configured to perform feature interaction on the obstacle features and the driving environment features to obtain interaction features. The prediction module 704 is configured to input the interaction features into a decoder to perform motion prediction and obtain a future behavior state of the obstacle around the vehicle. The obstacle and driving environment information includes surrounding perception information and surrounding map information of the vehicle. The driving environment features include stop line features and road features. The extraction module 702 is further configured to extract the obstacle features and the stop line features from the surrounding perception information, and extract the road features from the surrounding map information.

[0104] In the embodiment, in the obstacle behavior prediction device 700 based on feature interaction: the specific processing of the acquisition module 701, the extraction module 702, the interaction module 703, and the prediction module 704 and the technical effects brought by the specific processing can be respectively referred to the related descriptions of steps 101-104 in the corresponding embodiment, which will not be repeated here. Figure 1

[0105] ​In some optional implementations of the present embodiment, the obstacle and driving environment information includes surrounding perception information and surrounding map information of the vehicle, and the driving environment feature includes a stop line feature and a road feature.

[0106] In some optional implementations of the present embodiment, the extraction module 702 is further configured to extract the obstacle feature and the stop line feature from the surrounding perception information, and extract the road feature from the surrounding map information.

[0107] In some optional implementations of the present embodiment, the interaction module 703 includes an interaction sub-module configured to input the obstacle feature and the driving environment feature to an encoder for multi-level feature interaction to obtain the interaction feature.

[0108] In some optional implementations of the present embodiment, the interaction sub-module includes an embedding unit configured to perform feature embedding on the obstacle feature, the stop line feature and the road feature to obtain obstacle embedded features and road embedded features; and an interaction unit configured to perform multi-level feature interaction on the obstacle embedded features and the road embedded features using an attention mechanism to obtain the interaction feature.

[0109] In some optional implementations of the present embodiment, the interaction unit includes a convolution sub-unit configured to input the obstacle feature, the stop line feature and the road feature to corresponding convolution layers respectively to obtain obstacle intermediate features, stop line intermediate features and road intermediate features; and a concatenation sub-unit configured to concatenate the obstacle intermediate features and the stop line intermediate features to obtain the obstacle embedded features, and take the road intermediate features as the road embedded features.

[0110] In some optional implementations of the present embodiment, the number of convolution layers is proportional to the dimension of the input features of the convolution layers.

[0111] In some optional implementations of the present embodiment, the convolution sub-unit is further configured to input the obstacle feature and the road feature to corresponding multiple convolution layers in series respectively to obtain the obstacle intermediate features and the road intermediate features; and input the stop line feature to one convolution layer to obtain the stop line intermediate features.

[0112] In some optional implementations of the present embodiment, the interaction unit is further configured to perform feature interaction on the obstacle embedded features and the road embedded features using an attention mechanism to obtain a first interaction feature of the vehicle to the road; perform feature interaction on the first interaction feature using a self-attention mechanism to obtain a second interaction feature of the road to the road; perform feature interaction on the second interaction feature and the obstacle embedded features using an attention mechanism to obtain a third interaction feature of the road to the vehicle; and perform feature interaction on the third interaction feature using a self-attention mechanism to obtain a fourth interaction feature of the vehicle to the vehicle.

[0113] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solutions comply with relevant laws and regulations and do not violate public order and good customs.

[0114] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, a computer program product and an autonomous vehicle.

[0115] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0116] As shown in Figure 8 The device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0117] Various components in the device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, etc., an output unit 807, such as various types of displays, speakers, etc., the storage unit 808, such as a magnetic disk, an optical disk, etc., and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0118] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the feature interaction based obstacle behavior prediction method. For example, in some embodiments, the feature interaction based obstacle behavior prediction method can be implemented as a computer software program tangibly embodied in a machine readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the feature interaction based obstacle behavior prediction method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the feature interaction based obstacle behavior prediction method by any other suitable means, such as by means of firmware. The autonomous vehicle includes the electronic device 800.

[0119] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0120] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0121] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0122] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0123] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0124] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0125] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps deleted, using the steps. For example, the steps described in the present disclosure can be executed in parallel, in sequence, or in a different order, as long as the desired results of the technical solutions provided by the present disclosure can be achieved, which are not limited herein.

[0126] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for predicting behavior of an obstacle based on feature interaction, comprising obtaining an obstacle and driving environment information of a surrounding of a vehicle; extracting obstacle features and driving environment features from the obstacle and driving environment information; inputting the obstacle features and the driving environment features into an encoder for multi-level feature interaction to obtain interaction features; inputting the interaction features into a decoder for motion prediction to obtain a future behavior state of the obstacle in the surrounding of the vehicle; wherein the obstacle and driving environment information comprises surrounding perception information and surrounding map information of the vehicle, and the driving environment features comprise stop line features and road features; and the extracting obstacle features and driving environment features from the obstacle and driving environment information comprises: extracting the obstacle features and the stop line features from the surrounding perception information, and extracting the road features from the surrounding map information.

2. The method of claim 1, wherein, the inputting the obstacle features and the driving environment features into an encoder for multi-level feature interaction to obtain the interaction features comprises: performing feature embedding on the obstacle features, the stop line features and the road features to obtain obstacle embedding features and road embedding features; performing multi-level feature interaction on the obstacle embedding features and the road embedding features by using an attention mechanism to obtain the interaction features.

3. The method of claim 2, wherein, the performing feature embedding on the obstacle features, the stop line features and the road features to obtain obstacle embedding features and road embedding features comprises: inputting the obstacle features, the stop line features and the road features into corresponding convolution layers respectively to obtain obstacle intermediate features, stop line intermediate features and road intermediate features; concatenating the obstacle intermediate features and the stop line intermediate features to obtain obstacle embedding features, and taking the road intermediate features as the road embedding features.

4. The method of claim 3, wherein, a number of the convolution layers is proportional to a dimension of input features of the convolution layers.

5. The method of claim 4, wherein, the inputting the obstacle features, the stop line features and the road features into corresponding convolution layers respectively to obtain obstacle intermediate features, stop line intermediate features and road intermediate features comprises: inputting the obstacle features and the road features into corresponding multiple convolution layers in series respectively to obtain the obstacle intermediate features and the road intermediate features; inputting the stop line features into one convolution layer to obtain the stop line intermediate features.

6. The method of claim 2, wherein, the performing multi-level feature interaction on the obstacle embedding features and the road embedding features by using an attention mechanism to obtain the interaction features comprises: performing feature interaction on the obstacle embedding features and road embedding features by using an attention mechanism to obtain first interaction features of a vehicle to a road; performing feature interaction on the first interaction features by using a self-attention mechanism to obtain second interaction features of a road to a road; performing feature interaction on the second interaction features and the obstacle embedding features by using an attention mechanism to obtain third interaction features of a road to a vehicle; performing feature interaction on the third interaction features by using a self-attention mechanism to obtain fourth interaction features of a vehicle to a vehicle.

7. An obstacle behavior prediction apparatus based on feature interaction, comprising an acquisition module configured to acquire obstacle and driving environment information of a vehicle periphery; an extraction module configured to extract obstacle features and driving environment features from the obstacle and driving environment information; an interaction module configured to input the obstacle features and the driving environment features to an encoder for multi-level feature interaction to obtain interaction features; a prediction module configured to input the interaction features to a decoder for motion prediction to obtain a future behavior state of an obstacle in the vehicle periphery; wherein the obstacle and driving environment information comprises periphery perception information and periphery map information of the vehicle, and the driving environment features comprise stop line features and road features; and the extraction module is further configured to: extract the obstacle features and the stop line features from the periphery perception information, and extract the road features from the periphery map information.

8. The apparatus of claim 7, wherein, the interaction module comprises: an embedding unit configured to perform feature embedding on the obstacle features, the stop line features and the road features to obtain obstacle embedding features and road embedding features; an interaction unit configured to perform multi-level feature interaction on the obstacle embedding features and the road embedding features using an attention mechanism to obtain the interaction features.

9. The apparatus of claim 8, wherein, the interaction unit comprises: a convolution sub-unit configured to input the obstacle features, the stop line features and the road features to corresponding convolution layers respectively to obtain obstacle intermediate features, stop line intermediate features and road intermediate features; a concatenation sub-unit configured to concatenate the obstacle intermediate features and the stop line intermediate features to obtain obstacle embedding features, and to take the road intermediate features as the road embedding features.

10. The apparatus of claim 9, wherein, a number of the convolution layers is proportional to a dimension of input features of the convolution layers.

11. The apparatus of claim 10, wherein, the convolution sub-unit is further configured to: input the obstacle features and the road features to corresponding multiple convolution layers in series respectively to obtain the obstacle intermediate features and the road intermediate features; input the stop line features to one convolution layer to obtain the stop line intermediate features.

12. The apparatus of claim 8, wherein, the interaction unit is further configured to: perform feature interaction on the obstacle embedding features and road embedding features using an attention mechanism to obtain first interaction features of a vehicle to a road; perform feature interaction on the first interaction features using a self-attention mechanism to obtain second interaction features of a road to a road; perform feature interaction on the second interaction features and the obstacle embedding features using an attention mechanism to obtain third interaction features of a road to a vehicle; perform feature interaction on the third interaction features using a self-attention mechanism to obtain fourth interaction features of a vehicle to a vehicle.

13. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1-6.

15. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-6.

16. An autonomous vehicle comprising the electronic device of claim 13.

Images