A joint target detection and trajectory prediction method based on risk field feedback

Abstract

The application discloses a joint target detection and trajectory prediction method based on risk field feedback, which extracts target observation features in a scene by using a multi-source feature extraction network; target observation features are processed first to output category prediction results and bounding box parameter prediction results; then, the geometric occupancy risk of an agent, the relative distance and the relative speed of the agent from a vehicle are calculated based on the target detection results, time-space dynamic motion risks are constructed, and target type weight risks are calculated; then, the geometric occupancy risks of all agents, the time-space dynamic motion risks, and the target type weight risks are weighted and summed to obtain a risk field; target observation features are updated based on the risk field to obtain corrected target detection data; and trajectory prediction is performed based on the corrected target detection data and the risk field, so that the adverse effects of detection errors on prediction results can be effectively reduced, and the target detection accuracy, the trajectory prediction accuracy and the overall safety in a complex traffic scene are improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and autonomous driving technology, and relates to a joint environment perception and motion prediction method for complex traffic scenarios, and particularly to a joint target detection and trajectory prediction method based on risk field feedback. Background Technology

[0002] With the continuous development of autonomous driving technology, object detection and trajectory prediction have become key technical components in autonomous driving systems. Object detection is primarily used to identify the location, category, and motion state of surrounding traffic participants, while trajectory prediction estimates the future motion trends of traffic participants based on historical motion information and scene environment information, thus providing forward-looking environmental information support for subsequent decision-making, planning, and control. Most existing autonomous driving systems adopt a serial processing approach of "environmental perception—motion prediction—decision-planning," with object detection and trajectory prediction typically built as independent functional modules. While this approach has the advantages of clear structure and relatively simple implementation, it also has certain limitations: on the one hand, errors in object detection results are easily propagated to the trajectory prediction module in subsequent processing, affecting the accuracy of the prediction results; on the other hand, motion trend information, interaction information, and potential risk information generated during trajectory prediction are difficult to influence the object detection module, resulting in a lack of effective information feedback and closed-loop collaboration between modules. Therefore, there is an urgent need to propose a joint object detection and trajectory prediction method based on risk field feedback to achieve synergistic optimization between object detection, trajectory prediction, and risk assessment, thereby improving detection accuracy, prediction accuracy, and overall safety in complex traffic scenarios. Summary of the Invention

[0003] In view of the problems in existing autonomous driving systems, such as the independence of target detection and trajectory prediction, the easy transmission of detection errors to the prediction module, and the difficulty in incorporating scene risk information into the collaborative optimization of front-end perception and prediction, this invention proposes a joint target detection and trajectory prediction method based on risk field feedback. This method is designed for complex traffic scenarios and constructs a closed-loop processing framework integrating target detection, trajectory prediction, and risk assessment. By introducing the scene risk field into the target detection and trajectory prediction process, it achieves collaborative modeling of the environmental target state and future motion trends. This approach not only enhances the target detection's ability to perceive key targets and high-risk areas but also improves the trajectory prediction's ability to characterize complex interactive behaviors and dangerous evolution trends. Compared with existing serial processing methods, this invention can effectively reduce the adverse effects of detection errors on prediction results and improve the target detection accuracy, trajectory prediction accuracy, and overall safety in complex traffic scenarios.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A joint target detection and trajectory prediction method based on risk field feedback, the method includes the following steps:

[0006] Step 1. Collect multi-source environmental perception data of autonomous vehicles in road traffic environment, preprocess and align the data in time and space, and then use multi-source feature extraction network to extract target observation features in the scene;

[0007] Step 2. Use MLP The target observation features are then nonlinearly mapped, concatenated with map features, and then used... MLP Nonlinear mapping is performed to obtain initial map detection features; then, the features are encoded by a Transformer encoding layer, fused at multiple scales by a feature pyramid network, and finally output by a classification subnetwork and a parameter regression subnetwork to produce category prediction results and bounding box parameter prediction results. The two are then concatenated to obtain the target detection result.

[0008] Step 3. Based on the target detection results, calculate the geometric occupancy risk of the agent and its relative distance and speed with the autonomous vehicle to construct the spatiotemporal dynamic motion risk between the agent and the autonomous vehicle; calculate the target type weight risk based on the target detection results and the spatiotemporal dynamic motion risk; then, sum the weighted geometric occupancy risks, spatiotemporal dynamic motion risks, and target type weight risks of all agents to obtain the risk field at time t; obtain and stack all risk fields of historical time steps, perform nonlinear mapping using MLP, and input them into a long short-term memory network for temporal modeling to obtain the temporal features of the risk field;

[0009] Step 4. After concatenating the target observation features with the risk field temporal features, use a linear mapping layer to map them to the original dimensions of the target observation features and update the target observation features; then repeat step 2 to update and obtain the corrected target detection data;

[0010] Step 5. Perform trajectory prediction based on the corrected target detection data and risk field.

[0011] As a preferred embodiment of the present invention, the multi-source environmental perception data mentioned in step 1 includes multi-view camera images, lidar point clouds, millimeter-wave radar data, and vehicle motion state data; the multi-view camera images include at least one or more view images from the vehicle's forward, rear, left, and right view directions; the vehicle motion state data includes one or more of the vehicle's position, speed, acceleration, heading angle, and yaw rate.

[0012] As a preferred embodiment of the present invention, the preprocessing operations in step 1 include image distortion correction, point cloud denoising, radar clutter filtering, outlier data removal, and missing data completion; when extracting target observation features in the scene, the input image data is processed using a visual Transformer network to obtain image features; the input point cloud data is processed using a convolutional neural network to obtain point cloud features; the input radar data is processed using a recurrent neural network (GRU) to obtain radar features; and the obtained image features, point cloud features, and radar features are used to process the input autonomous vehicle motion data to obtain autonomous vehicle motion compensation information.

[0013] As a preferred embodiment of the present invention, the classification subnetwork in step 2 is composed of a one-dimensional convolutional neural network, used to output the category and probability of the agent. The categories are specifically divided into three types: pedestrians, cyclists, and vehicles. The parametric regression subnetwork is composed of a three-layer MLP, used to output the three-dimensional coordinates, length, width, height, heading angle, and velocity of the agent. The MLP consists of two linear mapping layers, a ReLU activation function, and a Dropout layer. The map features are obtained by first acquiring high-precision map information, and then using a piecewise linear encoder to encode the map features.

[0014] As a preferred embodiment of the present invention, the geometric occupancy risk of agent i at time t in step 3 is... Represented as:

[0015] ;

[0016] In the formula, To detect confidence, i.e., class probability; This represents the x and y coordinates of the autonomous vehicle at time t. This represents the x and y coordinates of agent i at time t. express The inverse matrix, The covariance matrix is ​​related to the target's length, width, and orientation. ; Let i be the heading angle of agent i at time t. The generated rotation matrix, These are the target length and width, respectively. The scaling parameters are set to 0.4 and 0.6. This is the transpose symbol.

[0017] As a preferred embodiment of the present invention, the spatiotemporal dynamic motion risk of the intelligent agent i and the autonomous vehicle at time t in step 3 is... Represented as:

[0018] ;

[0019] In the formula, This is the distance attenuation coefficient, with a value of 0.6. The speed risk amplification factor is set to 0.4. To test the confidence level, Let be the relative distance between agent i and the autonomous vehicle at time t. Let be the relative speed between agent i and the autonomous vehicle at time t. The geometric occupancy risk of agent i at time t;

[0020] At time t, the target type weight risk of intelligent agent i and autonomous vehicle Represented as:

[0021] ;

[0022] In the formula, The target type weight risk factor is determined by the type of agent: pedestrians are set to 0.7, cyclists to 0.5, and vehicles to 0.3.

[0023] As a preferred embodiment of the present invention, the risk field at time t in step 3 Represented as:

[0024] ;

[0025] In the formula, The weighting factors are set to 0.3, 0.6, and 0.1. Let be the geometric occupancy risk of agent i at time t. Let the target type weight risk of agent i and autonomous vehicle at time t be denoted by t. Let N represent the spatiotemporal dynamic motion risk of agent i and the autonomous vehicle at time t, and let N represent the total number of agents in the scenario.

[0026] As a preferred embodiment of the present invention, step 5 specifically includes the following steps:

[0027] Step 5.1. Reassemble the corrected target detection data to obtain the agent's historical trajectory. Types of intelligent agents ;

[0028] Step 5.2. Analyze the historical trajectory of the agent obtained in Step 5.1. Temporal modeling was performed using bidirectional gated cyclic units to obtain the temporal features of historical trajectories. ;

[0029] Step 5.3. Utilize the historical trajectory temporal features obtained in Step 5.2. The type of agent in step 5.1 By concatenating the features, we obtain the initial features of all agents in the scene. ;

[0030] Step 5.4. Utilize the initial features of all agents obtained in Step 5.3. A multi-head self-attention mechanism is used to capture the interactions of the agent and obtain the agent's interaction features. ;

[0031] Step 5.5. Combine map features with agent interaction features from Step 5.4. By stitching together the data, we obtain the intelligent agent interaction and map stitching features. ;

[0032] Step 5.6. Combine the agent interaction and map stitching features obtained in Step 5.5. Using MLP for mapping, we obtain the initial interaction features. ;

[0033] Step 5.7. Use the Transformer encoding layer to process the initial interaction features obtained in Step 5.6. Encode the interaction features F;

[0034] Step 5.8. Concatenate the interaction feature F with the risk field temporal feature to obtain... Subsequently, a multi-head self-attention mechanism was used for fusion to obtain the decoded features. ;

[0035] Step 5.9. Based on the decoding features in Step 5.8 We use a three-layer MLP to decode the future trajectory of the agent in the scenario.

[0036] As a further preferred embodiment of the present invention, step 5.1 extracts the three-dimensional coordinates, heading angle, and velocity of the agent from the corrected target detection data to construct the state of agent i at time t. And the historical trajectories of N intelligent agents in the scene at time t. The organization is: The corrected target detection data, including the length, width, height, and type of the agent, is used to construct the type attribute of agent i. And organize the types of N intelligent agents in the scene as follows: , .

[0037] As a further preferred embodiment of the present invention, in step 5.4, the multi-head self-attention mechanism uses 8 attention heads and the number of attention layers is set to 1; in step 5.8, the multi-head self-attention mechanism uses 8 attention heads and the number of attention layers is set to 3.

[0038] Advantages and beneficial effects of the present invention:

[0039] (1) This invention addresses the problem in existing autonomous driving systems where target detection and trajectory prediction are independent and detection errors tend to accumulate and propagate to the prediction module. It proposes a joint target detection and trajectory prediction method based on risk field feedback. By constructing a scene risk field based on target detection results and feeding the risk information back to the target detection module, enhanced perception of high-risk areas and key targets can be achieved. This effectively reduces missed and false detections in complex traffic scenarios, thereby improving the input quality and overall accuracy of subsequent trajectory prediction.

[0040] (2) This invention constructs a scene risk field model that integrates target category, detection confidence, spatial location, geometric dimensions, heading angle, and speed information, which can comprehensively characterize the current scene risk from multiple dimensions such as geometric occupancy, dynamic motion, and traffic semantics. Compared with methods that only perform simple assessments based on distance or speed, the risk field constructed by this invention can more accurately reflect the potential impact of traffic participants on autonomous vehicles and the surrounding environment, thereby improving the refinement level of risk representation and scene adaptability.

[0041] (3) By introducing risk field feedback into the target detection module, this invention enables dynamic correction of detection features, detection attention areas, or detection results. This allows the target detection process to no longer rely solely on the original observation information at the current moment, but to prioritize and enhance key targets based on the risk distribution of the scene. This mechanism helps improve the detection stability of occluded targets, small targets at long distances, and high-risk interactive targets, thereby enhancing the environmental perception capability of autonomous driving systems in complex traffic environments.

[0042] (4) This invention not only uses the risk field for target detection enhancement, but also introduces the risk field into the trajectory prediction module, so that the trajectory prediction process can be jointly modeled by scene risk distribution, high-risk interaction relationship and potential conflict trend, so as to suppress high-risk trajectory modes and enhance low-risk reasonable trajectories, thereby making the prediction results more in line with the real traffic evolution law and safety constraints, thereby improving the rationality, stability and safety of multimodal trajectory prediction.

[0043] (5) This invention optimizes target detection and trajectory prediction within a unified framework, enabling risk field feedback to simultaneously affect both perception and prediction stages. This reduces the cascading amplification effect of detection errors on the trajectory prediction module from a methodological perspective. Compared to the traditional serial "detection-prediction" processing method, this invention achieves collaborative optimization between detection and prediction, effectively reducing cumulative errors and improving the continuity and accuracy of overall environmental understanding.

[0044] (6) Under the combined effect of risk field feedback and joint optimization mechanism, this invention can not only improve the reliability of target detection, but also enhance the ability of trajectory prediction to characterize complex interactive behaviors, dangerous evolution trends and abnormal traffic actions. Especially in high-risk scenarios such as sudden lane changes, emergency deceleration, occlusion and crossing, and dense game, this invention can demonstrate stronger risk perception ability, detection robustness and trajectory prediction stability.

[0045] (7) The method of the present invention is applicable to various complex traffic scenarios such as urban roads, highways, intersections and vehicle merging and merging, and can be deployed on existing autonomous driving hardware platforms. It mainly relies on conventional vehicle perception data such as multi-view cameras, lidar, millimeter-wave radar, positioning units and vehicle motion state information, without the need to add additional high-cost external perception equipment. It has good engineering feasibility, system compatibility and promotion application value. Attached Figure Description

[0046] Other objects and results of the invention will become more apparent and readily understood with reference to the following description taken in conjunction with the accompanying drawings. In the drawings:

[0047] Figure 1 The flowchart shows the joint target detection and trajectory prediction method based on risk field feedback provided by this invention. Detailed Implementation

[0048] To enable those skilled in the art to better understand the technical solutions and advantages of the present invention, the present application will be described in detail below with reference to the accompanying drawings, but this is not intended to limit the scope of protection of the present invention.

[0049] like Figure 1 As shown, this embodiment provides a joint target detection and trajectory prediction method based on risk field feedback. This method first acquires the spatial location, category information, and motion state of traffic participants based on vehicle surrounding environment perception data, and then combines multi-view image features, temporal features, or scene context information to complete target detection at the current moment. Based on the target detection results and the relative positional relationships, motion relationships, and potential conflict relationships among traffic participants, a risk field reflecting the level of danger and risk evolution trend of the scene is constructed, and risk characterization information is extracted. Further, the risk characterization information is fed back to the target detection module and the trajectory prediction module to correct and constrain the detection results, target feature expression, trajectory modal weights, or future trajectory generation process, thereby forming a closed-loop joint processing mechanism of "target detection—risk assessment—trajectory prediction—feedback optimization".

[0050] Specifically, the joint target detection and trajectory prediction method based on risk field feedback described in this embodiment includes the following steps:

[0051] Step 1. Data Acquisition and Feature Extraction:

[0052] Step 1.1. Multi-source environmental sensing data acquisition:

[0053] Collect multi-source environmental perception data of autonomous vehicles in road traffic environments, including multi-view camera images. LiDAR point cloud Millimeter-wave radar data Vehicle's own motion status data ;in, The multi-view camera images represent time t. The lidar point cloud data at time t, Millimeter-wave radar data at time t This represents the vehicle's own motion state data at time t. .

[0054] In this embodiment, the multi-view camera images include at least one or more view images from the vehicle's front, rear, left, and right directions, used to characterize the visual semantic information of the current scene; the lidar point cloud is used to characterize the three-dimensional spatial distribution information of the target and environment in the scene; the millimeter-wave radar data is used to characterize the target's distance, relative speed, and orientation information; and the vehicle's own motion state data includes one or more of the vehicle's position, speed, acceleration, heading angle, and yaw rate.

[0055] Step 1.2. Sensitive data preprocessing and spatiotemporal alignment:

[0056] Based on the multi-view camera images obtained in step 1.1 LiDAR point cloud Millimeter-wave radar data Vehicle's own motion status data Preprocessing and spatiotemporal alignment are performed to eliminate differences in sampling frequency, timestamps, and coordinate systems among different sensors, thereby constructing a unified scene representation. .

[0057] Specifically, it can be expressed as:

[0058] Let the raw multi-source sensing data at time t be... for:

[0059]

[0060] In this embodiment, the preprocessing operations include image distortion correction, point cloud denoising, radar clutter filtering, outlier removal, and missing data completion. The standardized sensing data after preprocessing... for:

[0061]

[0062] In the formula, This represents a mapping function that performs preprocessing operations on raw multi-source sensing data.

[0063] Furthermore, the data from all sensors are uniformly transformed to the coordinate system of the autonomous vehicle for spatiotemporal alignment:

[0064]

[0065] In the formula, Represents the coordinate transformation matrix. This represents the data at time t after preprocessing and spatiotemporal alignment.

[0066] For all sensors in the historical time window The alignment results within the sequence are uniformly organized to obtain a spatiotemporally aligned multi-source scene input sequence. :

[0067]

[0068] In the formula, , .

[0069] Step 1.3. Scene Feature Extraction:

[0070] Based on the spatiotemporally aligned multi-source scene input sequence obtained in step 1.2 Multi-source feature extraction networks are used to extract target observation features in a scene. This provides input for subsequent object detection; the expression is:

[0071]

[0072] In the formula, This represents a multi-source feature extraction network, composed of a visual Transformer network, a convolutional neural network, and a recurrent neural network (GRU). It represents the set of target observation features consisting of image features, point cloud features, radar features, and motion compensation information from autonomous vehicles.

[0073] Specifically, in this embodiment, a visual Transformer network is used to process the input image data to obtain image features; a convolutional neural network is used to process the input point cloud data to obtain point cloud features; a recurrent neural network (GRU) is used to process the input radar data to obtain radar features; and the obtained image features, point cloud features, and radar features are used to process the input autonomous vehicle motion data to obtain autonomous vehicle motion compensation information.

[0074] Step 2. Target Detection:

[0075] Step 2.1. Based on the target observation features generated in Step 1.3 Using a simple MLP Perform nonlinear mapping to obtain initial detection features. The expression is:

[0076]

[0077] In the formula, the MLP consists of two linear mapping layers, a ReLU activation function, and a Dropout layer.

[0078] Step 2.2. Obtain high-precision map information M Map features are obtained by encoding them using a polyline encoder. , can be represented as:

[0079]

[0080] In the formula, This represents a piecewise linear encoder.

[0081] Step 2.3. Utilize the initial detection features obtained in Step 2.1 Map features obtained in step 2.2 After stitching, a simple MLP is used for non-linear mapping to obtain the initial map detection features. , can be represented as:

[0082]

[0083]

[0084] In the formula, This indicates a splicing operation. Representing the concatenated features, the MLP consists of two linear mapping layers, a ReLU activation function, and a Dropout layer.

[0085] Step 2.4. Detect features based on the initial map from Step 2.3. Encoding is performed using a Transformer encoding layer to obtain enhanced detection features. , can be represented as:

[0086]

[0087] In the formula, This represents the Transformer coding layer, which consists of a multi-head self-attention module, a feedforward neural network module, a residual connection module, and a normalization module.

[0088] Step 2.5. Enhanced detection features based on Step 2.4 Multi-scale feature fusion is performed using a Feature Pyramid Network (FPN) to obtain fused detection features. , can be represented as:

[0089]

[0090] In the formula, This represents the Feature Pyramid Network (FPN).

[0091] Step 2.6. Based on the fusion detection features obtained in Step 2.5 Use a classification subnetwork to output the category prediction result. The bounding box parameter prediction results are output using a parametric regression subnetwork. , can be represented as:

[0092]

[0093]

[0094] In the formula, The classification subnetwork is composed of a one-dimensional convolutional neural network. The parametric regression subnetwork consists of three simple MLP layers, each consisting of two linear mapping layers, a ReLU activation function, and a Dropout layer.

[0095] Specifically, ;in, and represent the category and probability (detection confidence) of the Nth agent in the scene at time t, respectively. In this embodiment, the agent categories are specifically divided into 3 categories: pedestrian, cyclist, and vehicle (car).

[0096]

[0097] in, Let (x, y, z) represent the coordinates of the Nth agent in the scene at time t. Let (l, w, h) represent the length, width, and height of the Nth agent in the scene at time t. Let be the heading angle of the Nth agent in the scene at time t. Let t be the velocity of the Nth agent in the scene at time t.

[0098] Step 2.7. Calculate the category prediction results obtained in Step 2.6. And bounding box parameter prediction results By stitching the data together, the final target detection result is obtained. , can be represented as:

[0099]

[0100] In the formula, This indicates a concatenation operation, which concatenates the feature vectors along the first dimension. Let N represent the set of target detection results at time t, and let N represent the total number of agents in the scene. This represents the detection result of the i-th target at time t. .

[0101] This invention trains the target detection module, enabling it to perform target detection at the current moment based on currently collected multi-source environmental perception data.

[0102] Step 3. Risk Field Construction:

[0103] Step 3.1. Utilize the target detection results from Step 2.7 Geometric occupancy risk of agent i at time t , can be represented as:

[0104]

[0105] In the formula, To test the confidence level, This represents the x and y coordinates of the autonomous vehicle at time t. This represents the x and y coordinates of agent i at time t. It is the transpose symbol. express The inverse matrix, The covariance matrix related to the target's length, width, and orientation:

[0106]

[0107] In the formula, Let i be the heading angle of agent i at time t. The generated rotation matrix, These are the target length and width, respectively. Set the scale adjustment parameters to 0.4 and 0.6.

[0108]

[0109] Step 3.2. Utilize the target detection results from Step 2.7 Calculate the relative distance between agent i and the autonomous vehicle at time t. and relative velocity Based on the geometric occupancy risk at time t obtained in step 3.1 And the relative distance between agent i and the autonomous vehicle at time t. and relative velocity Constructing the spatiotemporal dynamic motion risk of intelligent agent i and autonomous vehicle at time t , can be represented as:

[0110] The relative distance between agent i and autonomous vehicle at time t :

[0111]

[0112] The relative speed between agent i and autonomous vehicle at time t :

[0113]

[0114] In the formula, This represents the velocity of agent i at time t. This represents the speed of the autonomous vehicle at time t.

[0115] Spatiotemporal dynamic motion risks of intelligent agent i and autonomous vehicle at time t:

[0116]

[0117] In the formula, This is the distance attenuation coefficient, with a value of 0.6. The speed risk amplification factor is set to 0.4. To test the confidence level.

[0118] Step 3.3. Utilize the target detection results from Step 2.7 The spatiotemporal dynamic motion risk of agent i and autonomous vehicle at time t obtained in step 3.2. Calculate target type weight risk , can be represented as:

[0119]

[0120] In the formula, The target type weight risk factor is determined by the type of the agent. The decision is as follows: pedestrians 0.7, cyclists 0.5, and vehicles 0.3.

[0121] Step 3.4. Using the methods in Steps 3.1, 3.2, and 3.3, calculate the geometric occupancy risk, spatiotemporal dynamic motion risk, and target type weight risk of all agents in the scenario at time t, and then sum them up by weight to obtain the risk field at time t. , can be represented as:

[0122]

[0123] In the formula, The weighting factors are set to 0.3, 0.6, and 0.1.

[0124] Step 3.5. Obtain historical time steps All risk fields are stacked into a single risk field vector. .

[0125] Step 3.6. Use a simple MLP to analyze the risk field vector. By performing nonlinear mapping, high-dimensional risk field characteristics are obtained. , can be represented as:

[0126]

[0127] In the formula, the MLP consists of two linear mapping layers, a ReLU activation function, and a Dropout layer.

[0128] Step 3.7. Use a temporal LSTM network to process the high-dimensional risk field features from Step 3.6. Temporal modeling is performed to obtain the temporal characteristics of the risk field. , can be represented as:

[0129]

[0130] In the formula, LSTM represents Long Short-Term Memory Network.

[0131] Step 4. Update target detection data based on risk field feedback:

[0132] Step 4.1. Analyze the target observation features generated in Step 1.3. Compared with the risk field time series characteristics obtained in step 3.7 By splicing the data, the target observation risk characteristics are obtained. , can be represented as:

[0133]

[0134] Step 4.2. Observe the target risk characteristics based on Step 4.1 A linear mapping layer is used to map it back to the original target observation features. The dimensions are updated, and the target observation features are updated;

[0135] Step 4.3. Repeat all operations in Step 2 to update the corrected target detection data.

[0136] Step 5. Perform trajectory prediction based on the corrected target detection data and risk field:

[0137] Step 5.1. Data Preparation and Processing:

[0138] Reassemble the corrected target detection data from step 4.3 to obtain the historical trajectory of the intelligent agent (traffic participant). Types of intelligent agents (traffic participants) .

[0139] Specifically, in this embodiment, the modified The three-dimensional coordinates, heading angle, and velocity of the agent are used to construct the state of agent i (traffic participant) at time t: And organize the historical trajectories of the N agents in the scene at time t as follows: ;Will The length, width, height, and type of the intelligent agent are used to construct the type attributes of traffic participant i. The types of the N traffic participants (agents) in the scene are organized as follows: .

[0140] Step 5.2. Analyze the historical trajectory of the agent obtained in Step 5.1. Temporal modeling was performed using a bidirectional GRU (gated cyclic unit) to obtain the temporal features of the historical trajectory. , can be represented as:

[0141]

[0142] In the formula, It consists of three bidirectional GRUs.

[0143] Step 5.3. Utilize the historical trajectory temporal features obtained in Step 5.2. The type of agent in step 5.1 By concatenating the features, we obtain the initial features of all agents in the scene. , can be represented as:

[0144]

[0145] In the formula, This represents the initial characteristics of the agent.

[0146] Step 5.4. Utilize the initial features of all agents obtained in Step 5.3. A multi-head self-attention mechanism is used to capture the agent's interactions, thereby obtaining the agent's interaction features. , can be represented as:

[0147]

[0148] In the formula, It is a multi-head self-attention mechanism, with 8 attention heads and 1 attention layer.

[0149] Step 5.5. Transfer the map features from Step 2.2. Interaction features with the agent in step 5.4 By stitching together the data, we obtain the intelligent agent interaction and map stitching features. , can be represented as:

[0150]

[0151] In the formula, This represents the characteristics of intelligent agent interaction and map stitching.

[0152] Step 5.6. Combine the agent interaction and map stitching features obtained in Step 5.5. Using a simple MLP for mapping, we obtain the initial interaction features. , can be represented as:

[0153]

[0154] In the formula, Representing the initial interaction features, the MLP consists of two linear mapping layers, a ReLU activation function, and a Dropout layer.

[0155] Step 5.7. Use the Transformer encoding layer from Step 2.4 to process the initial interaction features obtained in Step 5.6. Encoding yields the interaction feature F, which can be represented as:

[0156]

[0157] In the formula, F represents the interaction characteristics between the agent and the map. This indicates the Transformer encoding layer.

[0158] Step 5.8. Combine the interaction feature F with the risk field temporal features from Step 3.7. spliced ​​together Then, a multi-head self-attention mechanism is used for fusion to obtain the decoded features. , can be represented as:

[0159]

[0160]

[0161] In the formula, It is a multi-head self-attention mechanism, with 8 attention heads and 3 attention layers, used to extract more refined features.

[0162] Step 5.9. Based on the decoding features in Step 5.8 Using a three-layer MLP to decode the future trajectory Y of an agent (traffic participant) in a scenario, it can be represented as:

[0163]

[0164] In the formula, It consists of three simple multilayer perceptrons (MLPs), each consisting of two linear mapping layers, a ReLU activation function, and a Dropout layer.

[0165] The present invention also provides an electronic device, comprising: one or more processors and a memory; wherein the memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned joint target detection and trajectory prediction method based on risk field feedback.

[0166] The present invention also provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned joint target detection and trajectory prediction method based on risk field feedback.

[0167] Those skilled in the art will understand that all or part of the functions of the various methods / modules in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the above functions can be implemented by executing the program with a computer. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented.

[0168] In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the programs can also be stored in storage media such as servers, other computers, disks, optical discs, flash drives, or portable hard drives. They can be downloaded or copied to the memory of the local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.

[0169] The above-described specific examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A joint target detection and trajectory prediction method based on risk field feedback, characterized in that, The method includes the following steps: Step 1. Collect multi-source environmental perception data of autonomous vehicles in road traffic environment, preprocess and align the data in time and space, and then use multi-source feature extraction network to extract target observation features in the scene; Step 2. Use MLP The target observation features are then nonlinearly mapped, concatenated with map features, and then used... MLP Nonlinear mapping is performed to obtain initial map detection features; then, the features are encoded by a Transformer encoding layer, fused at multiple scales by a feature pyramid network, and finally output by a classification subnetwork and a parameter regression subnetwork to produce category prediction results and bounding box parameter prediction results. The two are then concatenated to obtain the target detection result. Step 3. Based on the target detection results, calculate the geometric occupancy risk of the agent and its relative distance and speed with the autonomous vehicle to construct the spatiotemporal dynamic motion risk between the agent and the autonomous vehicle; calculate the target type weight risk based on the target detection results and the spatiotemporal dynamic motion risk; then, sum the weighted geometric occupancy risks, spatiotemporal dynamic motion risks, and target type weight risks of all agents to obtain the risk field at time t; obtain and stack all risk fields of historical time steps, perform nonlinear mapping using MLP, and input them into a long short-term memory network for temporal modeling to obtain the temporal features of the risk field; Step 4. After concatenating the target observation features with the risk field temporal features, use a linear mapping layer to map them to the original dimensions of the target observation features and update the target observation features; then repeat step 2 to update and obtain the corrected target detection data; Step 5. Perform trajectory prediction based on the corrected target detection data and risk field.

2. The joint target detection and trajectory prediction method based on risk field feedback according to claim 1, characterized in that, The multi-source environmental perception data mentioned in step 1 includes multi-view camera images, LiDAR point clouds, millimeter-wave radar data, and vehicle motion state data; the multi-view camera images include at least one or more view images from the vehicle's forward, rear, left, and right directions; the vehicle motion state data includes one or more of the vehicle's position, speed, acceleration, heading angle, and yaw rate.

3. The joint target detection and trajectory prediction method based on risk field feedback according to claim 1, characterized in that, The preprocessing operations in step 1 include image distortion correction, point cloud denoising, radar clutter filtering, outlier data removal, and missing data completion. When extracting target observation features in the scene, a visual Transformer network is used to process the input image data to obtain image features; a convolutional neural network is used to process the input point cloud data to obtain point cloud features; a recurrent neural network (GRU) is used to process the input radar data to obtain radar features; and the obtained image features, point cloud features, and radar features are used to process the input autonomous vehicle motion data to obtain autonomous vehicle motion compensation information.

4. The joint target detection and trajectory prediction method based on risk field feedback according to claim 1, characterized in that, The classification subnetwork described in step 2 consists of a one-dimensional convolutional neural network, used to output the agent's category and probability. The categories are specifically divided into three types: pedestrians, cyclists, and vehicles. The parametric regression subnetwork consists of a three-layer MLP, used to output the agent's three-dimensional coordinates, length, width, height, heading angle, and velocity. The MLP consists of two linear mapping layers, a ReLU activation function, and a Dropout layer. The map features are obtained by first acquiring high-precision map information, and then using a polyline encoder to encode the map features.

5. The joint target detection and trajectory prediction method based on risk field feedback according to claim 1, characterized in that, Geometric occupancy risk of agent i at time t in step 3 Represented as: ； In the formula, To detect confidence, i.e., class probability; This represents the x and y coordinates of the autonomous vehicle at time t. This represents the x and y coordinates of agent i at time t. express The inverse matrix, The covariance matrix is ​​related to the target's length, width, and orientation. ; Let i be the heading angle of agent i at time t. The generated rotation matrix, These are the target length and width, respectively. The scaling parameters are set to 0.4 and 0.

6. This is the transpose symbol.

6. The joint target detection and trajectory prediction method based on risk field feedback according to claim 1, characterized in that, The spatiotemporal dynamic motion risk of agent i and autonomous vehicle at time t in step 3. Represented as: ; In the formula, This is the distance attenuation coefficient, with a value of 0.

6. The speed risk amplification factor is set to 0.

4. To test the confidence level, Let be the relative distance between agent i and the autonomous vehicle at time t. Let be the relative speed between agent i and the autonomous vehicle at time t. The geometric occupancy risk of agent i at time t; At time t, the target type weight risk of intelligent agent i and autonomous vehicle Represented as: ； In the formula, The target type weight risk factor is determined by the type of agent: pedestrians are set to 0.7, cyclists to 0.5, and vehicles to 0.

3.

7. The joint target detection and trajectory prediction method based on risk field feedback according to claim 1, characterized in that, Risk field at time t in step 3 Represented as: ； In the formula, The weighting factors are set to 0.3, 0.6, and 0.

1. Let be the geometric occupancy risk of agent i at time t. Let the target type weight risk of agent i and autonomous vehicle at time t be denoted by t. Let N represent the spatiotemporal dynamic motion risk of agent i and the autonomous vehicle at time t, and let N represent the total number of agents in the scenario.

8. The joint target detection and trajectory prediction method based on risk field feedback according to claim 1, characterized in that, Step 5 specifically includes the following steps: Step 5.

1. Reassemble the corrected target detection data to obtain the agent's historical trajectory. Types of intelligent agents ; Step 5.

2. Analyze the historical trajectory of the agent obtained in Step 5.

1. Temporal modeling was performed using bidirectional gated cyclic units to obtain the temporal features of historical trajectories. ; Step 5.

3. Utilize the historical trajectory temporal features obtained in Step 5.

2. The type of agent in step 5.1 By concatenating the features, we obtain the initial features of all agents in the scene. ; Step 5.

4. Utilize the initial features of all agents obtained in Step 5.

3. A multi-head self-attention mechanism is used to capture the interactions of the agent and obtain the agent's interaction features. ; Step 5.

5. Combine map features with agent interaction features from Step 5.

4. By stitching together the data, we obtain the intelligent agent interaction and map stitching features. ; Step 5.

6. Combine the agent interaction and map stitching features obtained in Step 5.

5. Using MLP for mapping, we obtain the initial interaction features. ; Step 5.

7. Use the Transformer encoding layer to process the initial interaction features obtained in Step 5.

6. Encode the interaction features F; Step 5.

8. Concatenate the interaction feature F with the risk field temporal feature to obtain... Subsequently, a multi-head self-attention mechanism was used for fusion to obtain the decoded features. ; Step 5.

9. Based on the decoding features in Step 5.8 We use a three-layer MLP to decode the future trajectory of the agent in the scenario.

9. The joint target detection and trajectory prediction method based on risk field feedback according to claim 8, characterized in that, Step 5.1 Extracts the three-dimensional coordinates, heading angle, and velocity of the agent from the corrected target detection data to construct the state of agent i at time t. And the historical trajectories of N intelligent agents in the scene at time t. The organization is: The corrected target detection data, including the length, width, height, and type of the agent, is used to construct the type attribute of agent i. And organize the types of N intelligent agents in the scene as follows: , .

10. The joint target detection and trajectory prediction method based on risk field feedback according to claim 8, characterized in that, In step 5.4, the multi-head self-attention mechanism uses 8 attention heads and the number of attention layers is set to 1; in step 5.8, the multi-head self-attention mechanism uses 8 attention heads and the number of attention layers is set to 3.

Images