A multimodal interactive end-to-end autonomous driving method, system and electronic device

Abstract

The application provides a kind of multimodal interactive end-to-end autonomous driving method, system and electronic equipment, the method comprises: obtaining multimodal perception data and ego state data, anchor point alignment multimodal sparse scene representation construction, risk sparse representation for planning task is established, and ego center geometry interaction modeling, based on planning decoding generation ego candidate planning trajectory and risk score and confidence estimation and safety constraint driven end-to-end joint optimization.The system takes multiway look-around image, laser radar point cloud and ego state data as input, realizes the synchronous modeling of surrounding traffic participant motion information and ego future planning trajectory through unified sparse scene intermediate representation, and outputs target planning result meeting safety, smoothness and executability requirement.The application can effectively enhance the environment understanding ability and planning decision-making ability of end-to-end autonomous driving system in lane changing, merging, intersection yielding and dense traffic flow scene.

Description

Technical Field

[0001] This invention relates to the field of road traffic safety technology, and more specifically, to a multimodal interactive end-to-end autonomous driving method, system, and electronic device. Background Technology

[0002] With the development of autonomous driving technology, how to accurately understand the surrounding environment and safely plan the vehicle's future trajectory in complex and dynamic traffic scenarios has become a key issue in intelligent driving systems. Existing autonomous driving methods mainly include two categories: traditional modular methods and end-to-end methods. Traditional modular methods typically break down environmental perception, motion prediction, and trajectory planning into multiple independent modules that are executed in series, which can easily lead to error accumulation. Although end-to-end methods can directly model the relationship between environmental input and planning output within a unified framework, they still suffer from insufficient scene representation capabilities and inadequate modeling of planning-related interactions in complex interactive scenarios.

[0003] Existing end-to-end autonomous driving methods generally suffer from the following shortcomings: First, the alignment and fusion between multi-view images and multi-modal data such as LiDAR point clouds are insufficient, making it difficult to maintain consistency between semantic and geometric information within a unified representation space. Second, all targets in the scene are usually treated with approximate equal weighting, lacking key target selection and risk enhancement mechanisms around the autonomous vehicle's planning tasks, making it difficult to highlight targets and areas that truly affect the vehicle's future trajectory. Third, there is insufficient modeling of geometric interaction relationships between the autonomous vehicle and surrounding traffic participants, such as relative position, speed, heading, conflict relationships, and historical movement trends, resulting in shortcomings in safety, stability, and executability of the planned trajectories generated in high-interaction scenarios such as lane changing, lane merging, and yielding at intersections.

[0004] Therefore, there is an urgent need for an end-to-end autonomous driving method that can perform unified sparse representation of multimodal environmental information, explicitly model key interaction relationships around the autonomous vehicle planning task, and complete trajectory selection in combination with safety constraints, so as to improve the accuracy, stability and safety of environmental understanding and planning decisions in complex dynamic traffic scenarios. Summary of the Invention

[0005] This invention addresses the technical problems existing in the prior art by providing a multimodal interactive end-to-end autonomous driving method, system, and electronic device. This invention can effectively enhance the environmental understanding and planning decision-making capabilities of end-to-end autonomous driving systems in scenarios such as lane changing, lane merging, intersection yielding, and dense traffic flow.

[0006] According to a first aspect of the present invention, a multimodal interactive end-to-end autonomous driving method is provided, comprising: Acquire multimodal perception data and vehicle status data; the multimodal perception data includes multi-path surround view image data and lidar point cloud data; Feature encoding is performed on the obtained image data and point cloud data to extract semantic features from the image modality and geometric features from the point cloud modality. An anchor-aligned multimodal shared sparse scene representation is constructed in a unified coordinate system. Based on the obtained shared sparse scene representation, the spatiotemporal relationship information of surrounding traffic participants and key road areas relative to the vehicle is extracted, and risk sparse representation and vehicle-centric geometric interaction model are established for planning tasks. The obtained sparse risk representation and the vehicle-centric geometric interaction representation are used to plan and decode to generate candidate planning trajectories for the vehicle, and risk scoring and confidence estimation are performed on the candidate planning trajectories. During the training phase, the construction of shared sparse scene representations, risk sparse modeling, vehicle-centric geometric interaction modeling, and planning decoding are incorporated into a unified end-to-end training framework for joint optimization. During the inference phase, the candidate planning trajectories are re-scored based on the jointly optimized model to meet safety constraints, and the target planning trajectory that satisfies the scenario constraints is output.

[0007] Based on the above technical solution, the present invention can also be improved as follows.

[0008] Optionally, the vehicle status data includes one or more of the following: vehicle position and attitude, speed, acceleration, heading angle, yaw rate, steering status, and historical trajectory information.

[0009] Optionally, after acquiring the multimodal perception data and the vehicle status data, the method further includes: Image data is scaled and brightness standardized; point cloud data is denoised, outliers are removed, and coordinates are transformed; vehicle status data is time-aligned and sequence-cached, thereby uniformly mapping multimodal inputs to the vehicle coordinate system or bird's-eye view coordinate system.

[0010] Optionally, the step of performing feature encoding on the obtained image data and point cloud data to extract semantic features from the image modality and geometric features from the point cloud modality includes: Feature encoding is performed on the obtained image data and point cloud data respectively to extract semantic features from the image modality and geometric features from the point cloud modality; A set of sparse scene anchor points is constructed under a unified coordinate system. The sparse scene anchor points are used to characterize key locations, candidate traffic participants and potential conflict areas in the road space. Image semantic features and point cloud geometric features are projected, sampled, and aggregated onto sparse scene anchor points; By sampling and fusing cross-modal features, a unified shared sparse representation unit is formed at each sparse anchor point, so that the shared sparse representation unit contains both semantic and geometric information.

[0011] Optionally, the step of extracting the spatiotemporal relationship information of surrounding traffic participants and key road areas relative to the vehicle based on the obtained shared sparse scene representation, and establishing a risk sparse representation and vehicle-centric geometric interaction model for planning tasks includes: Based on the obtained shared sparse scene representation, the spatiotemporal relationship information of surrounding traffic participants and key road areas relative to the vehicle is extracted, including relative position, relative speed, relative heading, lane ownership relationship, lateral encroachment relationship, potential conflict area occupancy status and historical movement change trend. Assess the impact of each traffic participant and spatial area on the future trajectory of vehicles, and assign higher risk weights to targets and areas with higher impact to form a risk-enhanced sparse representation for planning tasks. Using the current state of the vehicle and the planning query as the interaction center, the geometric relationship between the key targets in the risk-enhanced sparse representation and the vehicle is encoded to construct a vehicle-centric geometric interaction representation; By jointly modeling risk weights and geometric interactions, priority is given to key targets and key areas that have a real impact on the future trajectory of the vehicle, rather than treating all objects in the scene equally.

[0012] Optionally, assigning high-risk weights to targets and areas with a high degree of impact includes: The targets and areas are divided into high-risk, medium-risk, and low-risk targets. High-risk targets are vehicles that cut in at close range and potential pedestrians crossing the road. Medium-risk targets are nearby vehicles or dynamic targets to be observed that may intersect with the vehicle's trajectory. Low-risk targets are distant targets or non-critical static areas that are not strongly associated with the vehicle's movement path.

[0013] Optionally, the process of planning and decoding the obtained sparse risk representation and the vehicle-centric geometric interaction representation includes: Based on the vehicle's current motion state, the risk weights and interaction relationships of surrounding key targets, the candidate trajectory or target trajectory of the vehicle in the future time domain is output. During the trajectory generation process, the risk score, confidence score, or distribution parameters of the candidate trajectories are output to characterize the safety and feasibility of different planning results in the current traffic scenario. Prioritize retaining low-risk, highly feasible planning results, suppress high-risk trajectories that may lead to collisions, boundary violations, or significant oscillations, and output target planning trajectories that meet scenario constraints.

[0014] Optionally, the step of re-scoring the candidate planning trajectory based on the jointly optimized model to meet safety constraints and outputting the target planning trajectory that satisfies the scenario constraints includes: Construct a joint loss function that includes planning fitting loss and safety constraint loss. The planning fitting loss is used to constrain the positional deviation, orientation deviation or dynamic deviation between the predicted trajectory and the reference trajectory, and the safety constraint loss is used to constrain the planning result to meet the real driving safety requirements. Introduce risk-leading terms or interactive salience-leading terms to enhance sensitivity to risk objectives and conflict areas; After training, an end-to-end autonomous driving model is obtained. The autonomous driving model receives multimodal environment input and outputs the future trajectory of the vehicle that meets the requirements of safety, stability and executability.

[0015] According to a second aspect of the present invention, a multimodal interactive end-to-end autonomous driving system is provided, comprising: The multimodal input module is used to receive multi-channel surround view image data, LiDAR point cloud data, and vehicle status data at the current moment. The shared sparse scene representation construction module is used to set sparse anchor points in the vehicle coordinate system, and to perform anchor point-level alignment sampling and fusion of image semantic features and point cloud geometric features to form a unified shared sparse scene. The risk sparse representation and vehicle-centric geometric interaction module is used to extract the position, speed, heading, lane affiliation and potential conflict relationship of surrounding targets relative to the vehicle, and to perform risk weighting on the shared sparse scene. At the same time, it establishes geometric interaction relationship between the vehicle and key targets with the vehicle planning query as the center. The unified motion planning decoding and safety constraint screening module is used to simultaneously output the future trajectory patterns of surrounding targets and candidate planned trajectories of the vehicle based on shared interactive features, and to perform re-scoring and screening of candidate trajectories in combination with collision constraints, boundary constraints, smoothness constraints and dynamic constraints. The joint loss and phased training module is used to perform end-to-end optimization of the entire network using a multi-task loss function and a two-stage training strategy.

[0016] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, the processor being configured to implement the steps of a multimodal interactive end-to-end autonomous driving method when executing a computer program stored in the memory.

[0017] The technical effects and advantages of this invention are as follows: This invention provides a multimodal interactive end-to-end autonomous driving method, system, and electronic device. By constructing a multimodal sparse scene representation for autonomous driving tasks, it achieves collaborative modeling of image semantic information, point cloud geometric information, and vehicle state information within a unified feature space. Furthermore, by combining risk sparse representation with a vehicle-centric geometric interaction mechanism, it can more accurately identify targets and regions that have a critical impact on the vehicle's future trajectory. Through unified motion planning decoding and joint optimization of safety constraints, it improves the safety, stability, and executability of trajectory generation in complex traffic scenarios. Therefore, this invention can effectively enhance the environmental understanding and planning decision-making capabilities of end-to-end autonomous driving systems in lane changing, merging, intersection yielding, and dense traffic flow scenarios, possessing significant engineering application value and promotional significance. Attached Figure Description

[0018] Figure 1 A flowchart of a multimodal interactive end-to-end autonomous driving method provided in an embodiment of the present invention; Figure 2 A schematic diagram of the shared sparse scene representation construction module provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the risk sparse representation and vehicle center geometry interaction module and the unified motion planning decoding module provided in the embodiments of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Understandably, given the deficiencies in the background technology, this invention proposes a multimodal interactive end-to-end autonomous driving method, specifically as follows: Figure 1 As shown, it includes the following steps: Step 1: Acquire multimodal perception data and vehicle status data; The multimodal perception data includes multi-path surround view image data and LiDAR point cloud data; The vehicle status data includes one or more of the following: vehicle pose, speed, acceleration, heading angle, yaw rate, steering status, and historical trajectory information; In this embodiment, environmental data is collected synchronously by multimodal sensors deployed on the autonomous vehicle. Among them, multi-channel surround-view cameras are used to collect images of the road scene around the vehicle, lidar is used to collect three-dimensional point cloud data of the environment around the vehicle, and vehicle status sensors are used to collect the vehicle's motion status information, including vehicle speed, acceleration, heading angle, steering status, and historical trajectory information.

[0021] Specifically, images of the road scene surrounding the vehicle are captured by front-view, side-view, and rear-view cameras, covering scene elements such as lane lines, road boundaries, traffic signs, traffic lights, vehicles, pedestrians, non-motorized vehicles, and static obstacles. Three-dimensional point cloud information of the surrounding environment is acquired using LiDAR to characterize the road spatial structure, target 3D position, size contour, and depth distribution. Information obtained from one or more of the vehicle chassis sensors, inertial measurement units, or annotation information from autonomous driving datasets is used for subsequent autonomous vehicle center interactive modeling and planning query construction.

[0022] After acquiring the multimodal perception data and the vehicle status data, the process also includes: Multimodal data undergoes time synchronization and basic cleaning processing. Specifically, image data is scaled and brightness standardized, point cloud data is denoised, outlier removed, and coordinate transformed, and vehicle status data is time-aligned and cached. This process maps the multimodal inputs to either the vehicle coordinate system or the bird's-eye view coordinate system.

[0023] Step 2: Perform feature encoding on the obtained image data and point cloud data, extract semantic features from the image modality and geometric features from the point cloud modality, and construct an anchor-aligned multimodal shared sparse scene representation under a unified coordinate system; The step of performing feature encoding on the obtained image data and point cloud data to extract semantic features from the image modality and geometric features from the point cloud modality includes the following steps: Feature encoding was performed on the obtained image data and point cloud data respectively to extract semantic features from the image modality and geometric features from the point cloud modality. Among them, the image semantic features mainly represent the road topology, target category attributes and scene texture information, while the point cloud geometric features mainly represent the target spatial location, depth information, size and shape and local structure information. A set of sparse scene anchor points is constructed under a unified coordinate system. The sparse scene anchor points are used to represent key locations, candidate traffic participants, and potential conflict areas in the road space. The anchor points can be generated based on road geometric distribution, vehicle neighborhood space division, target candidate centers, or learnable queries. Image semantic features and point cloud geometric features are projected, sampled and aggregated onto sparse scene anchor points. Among them, image features are projected to the corresponding positions of anchor points through camera intrinsic and extrinsic parameters for semantic sampling, and point cloud features are extracted through anchor point neighborhood aggregation. Features of different modalities can be weighted and fused according to visibility, quality or confidence. By sampling and fusing cross-modal features, a unified shared sparse representation unit is formed at each sparse anchor point, which simultaneously contains semantic and geometric information.

[0024] Step 3: Based on the obtained shared sparse scene representation, extract the spatiotemporal relationship information of surrounding traffic participants and key road areas relative to the vehicle, and establish a risk sparse representation and vehicle-centric geometric interaction model for planning tasks. Based on the obtained shared sparse scene representation, the spatiotemporal relationship information of surrounding traffic participants and key road areas relative to the vehicle is extracted, and risk sparse representation and vehicle-centric geometric interaction modeling for planning tasks are established, including: Based on the obtained shared sparse scene representation, the spatiotemporal relationship information of surrounding traffic participants and key road areas relative to the vehicle is extracted, including relative position, relative speed, relative heading, lane ownership, lateral encroachment, potential conflict area occupancy, and historical movement change trends.

[0025] The impact of each traffic participant and spatial area on the future trajectory of the vehicle is assessed, and corresponding risk weights are assigned to each traffic participant and spatial area according to the impact. This allows targets and areas with stronger correlation to the future trajectory of the vehicle and higher potential conflict levels to obtain stronger feature responses in the risk sparse representation, so as to form a risk-enhanced sparse representation for planning tasks. Among them, high-risk objects can be vehicles cutting in at close range and potential pedestrians crossing the road; medium-risk objects can be nearby vehicles or dynamic targets to be observed that may intersect the trajectory; and low-risk objects can be distant targets or non-critical static areas with weak correlation to the vehicle's movement path.

[0026] Using the current status of the vehicle and the planning query as the interaction center, the key targets in the risk-enhanced sparse representation and the geometric relationship between the vehicle are encoded to construct a vehicle-centric geometric interaction representation; the geometric interaction includes at least one or more of the following relationship, passing relationship, cutting in relationship, lane changing relationship, yielding relationship and potential collision relationship.

[0027] By jointly modeling risk weights and geometric interactions, the model prioritizes key targets and key areas that have a real impact on the future trajectory of the vehicle, rather than treating all objects in the scene equally.

[0028] Step 4: The obtained sparse risk representation and the vehicle-centric geometric interaction representation are used for planning and decoding to generate candidate planning trajectories for the vehicle, and risk scoring and confidence estimation are performed on the candidate planning trajectories; It should be noted that the planning decoding module refers to a learnable network submodule used to map the sparse risk representation, the vehicle-centric geometric interaction representation, and the vehicle state information into candidate future planning trajectories for the vehicle. The planning decoding module includes a feature aggregation unit, a candidate trajectory generation unit, a risk scoring unit, and a confidence estimation unit. The feature aggregation unit is used to pool, attention-aggregate, or query the sparse risk token set and fuse it with the vehicle-centric geometric interaction representation and the vehicle state information. The candidate trajectory generation unit is used to generate one or more candidate vehicle planning trajectories within the future planning time domain. The risk scoring unit is used to evaluate the associated risks between each candidate planning trajectory and the risk target, key road areas, and potential conflict areas. The confidence estimation unit is used to output a feasibility or reliability score for each candidate planning trajectory. The planning decoding module can be implemented using a query-based Transformer decoding structure, an attention decoding network, a trajectory pattern query network, a feedforward neural network, or a combination of the above structures, and is not limited to any specific network structure.

[0029] The process of planning and decoding the obtained sparse risk representation and the vehicle-centric geometric interaction representation to generate candidate vehicle trajectories, and then performing risk scoring and confidence estimation on the candidate trajectories, includes: The obtained sparse risk representation and the vehicle-centric geometric interaction representation are jointly input into the planning and decoding module; The planning and decoding module outputs candidate trajectories or target trajectories for the vehicle in the future time domain based on the vehicle's current motion state, the risk weights of surrounding key targets, and their interaction relationships. The target trajectory can be represented as a sequence of position points, heading sequence, velocity sequence, or equivalent trajectory parameterization results at several future moments. During trajectory generation, the risk score, confidence score or distribution parameters of candidate trajectories are output simultaneously to characterize the safety and feasibility of different planning results in the current traffic scenario. For high-interaction scenarios such as lane changing, lane merging, and intersection yielding, the planning decoding module sorts and filters multiple candidate trajectories based on the potential behavioral changes of key targets, the passable area of ​​the vehicle, and the intensity of potential conflicts. Prioritize retaining low-risk, highly feasible planning results, suppress high-risk trajectories that may lead to collisions, boundary violations, or significant oscillations, thereby outputting target planning trajectories that meet scenario constraints.

[0030] Step 5: During the training phase, the construction of shared sparse scene representation, risk sparse modeling, vehicle-centric geometric interaction modeling, and planning decoding are incorporated into a unified end-to-end training framework for joint optimization. During the inference phase, the candidate planning trajectories are re-scored based on the jointly optimized model to meet safety constraints, and the target planning trajectory that satisfies the scenario constraints is output.

[0031] Joint Training Framework: This framework integrates multimodal scene representation construction, risk sparse modeling, vehicle-centric geometric interaction modeling, and planning / decoding into a unified end-to-end training framework for joint optimization. Specifically, it includes: A joint loss function is constructed, which includes planning fitting loss and safety constraint loss. The planning fitting loss is used to constrain the positional deviation, directional deviation or dynamic deviation between the predicted trajectory and the reference trajectory, and the safety constraint loss is used to constrain the planning result to meet the real driving safety requirements.

[0032] The safety constraint loss includes at least one or more of collision constraints, road boundary constraints, and trajectory smoothing constraints; wherein, collision constraints are used to suppress spatiotemporal overlap between the predicted trajectory and surrounding key targets, road boundary constraints are used to prevent the trajectory from going beyond the passable area, and trajectory smoothing constraints are used to suppress trajectory abrupt changes, sharp turns, and abnormal oscillations.

[0033] During joint training, risk-guided terms or interaction saliency-guided terms are introduced to enhance the model’s sensitivity to high-risk targets and conflict areas, and to prevent the model from pursuing only trajectory point fitting errors while ignoring safety requirements in real driving.

[0034] After training, an end-to-end autonomous driving model is obtained. During the inference phase, it directly receives multimodal environment inputs and outputs the future trajectory of the autonomous vehicle that meets the requirements of safety, stability and executability.

[0035] The following is a further explanation using specific embodiments. This invention provides a multimodal interactive end-to-end autonomous driving method for the field of automotive autonomous driving, specifically including the following steps: 1. Acquisition of multimodal perception data and vehicle status data; Multimodal perception data includes multi-view surround view image data and LiDAR point cloud data; the multi-view surround view image data is acquired by multiple cameras deployed around the vehicle. The multiple cameras include front-view, front-left, front-right, rear-left, rear-right and rear-view cameras, which are used to cover the road environment around the vehicle and provide semantic cues such as lane lines, road boundaries, traffic participants, traffic signs, traffic lights and static obstacles.

[0036] LiDAR point cloud data is used to provide three-dimensional geometric information about the environment around a vehicle, including the three-dimensional position, depth distribution, size profile, and local spatial structure of surrounding targets.

[0037] Vehicle status data includes one or more of the following: vehicle pose, velocity, acceleration, heading angle, yaw rate, steering status, and historical trajectory information. This vehicle status data can be obtained from one or more of the following: vehicle chassis sensors, inertial measurement unit, or annotation information from an autonomous driving dataset. Let's assume the current time... The vehicle's state vector is: (1) In the formula, This represents the current state vector of the vehicle. and This indicates the position of the vehicle in the local coordinate system or the vehicle-related coordinate system. Indicates the vehicle's heading angle; Indicates the vehicle's speed; This indicates the longitudinal acceleration of the vehicle; This indicates the yaw rate of the vehicle. This indicates the vehicle's steering state.

[0038] 2: Construction of multimodal shared sparse scene representations with anchor point alignment; Feature encoding is performed on image data and point cloud data, and cross-modal alignment and fusion are achieved through sparse anchor points in a unified coordinate system to obtain a multimodal shared sparse scene representation.

[0039] Image feature encoding is performed on the multi-path loop view image data at the current moment to obtain image semantic features: (2) In the formula, This represents the semantic features of the image extracted from multi-path loop image data. Represents an image coding network. This represents the multi-path surround view image data at the current moment. The image semantic features are used to characterize the road topology, lane lines, road boundaries, target category attributes, and scene context information.

[0040] Point cloud feature encoding is performed on the current lidar point cloud data to obtain the point cloud geometric features: (3) In the formula, This represents the geometric features of the point cloud extracted from lidar point cloud data. Represents a point cloud coding network. This represents the current moment's lidar point cloud data. The geometric features of the point cloud are used to characterize the target's three-dimensional position, size, orientation, depth distribution, and local spatial structure.

[0041] Construct a sparse set of anchor points in a unified coordinate system: (4) In the formula, Represents a sparse set of anchor points. Indicates the first A sparse anchor point, This indicates the number of sparse anchor points. These sparse anchor points are used to characterize key locations, candidate traffic participants, and potential conflict zones in the road space.

[0042] Based on the sparse anchor points, the image semantic features and point cloud geometric features are mapped to a unified coordinate system for anchor point-level alignment and fusion, resulting in the first... Shared Scenario Token: (5) In the formula, Indicates the first A multimodal shared scenario token corresponding to a sparse anchor point. This represents the anchor-level cross-modal fusion function. Indicates the first A sparse anchor point, Representing the semantic features of an image, It represents the geometric features of a point cloud.

[0043] The set of multimodal shared sparse scene representations at the current moment is composed of all shared scene tokens: (6) In the formula, This represents the set of multimodal shared sparse scene representations at the current moment. Indicates the first A shared scenario token, This indicates the number of sparse anchor points. The multimodal shared sparse scene representation simultaneously includes semantic information provided by the image modality and geometric information provided by the point cloud modality, and serves as input for subsequent risk sparse representation generation and vehicle-centric geometric interaction modeling.

[0044] In one alternative implementation, the anchor-level cross-modal fusion function This can be achieved through one or more of the following methods: image projection sampling, point cloud neighborhood aggregation, attention fusion, or gated fusion.

[0045] 3: Sparse representation of risk and geometric interaction modeling at the center of the vehicle; This is used to screen and enhance key traffic participants and key road areas related to autonomous vehicle planning tasks from shared sparse scene representations, and to establish geometric interaction relationships between autonomous vehicles and key objects.

[0046] For the Each shared scenario token is used to construct its risk context vector for the planning task: (7) In the formula, Indicates the first Risk context vector of a shared scenario token; This indicates the positional relationship of the i-th target or region relative to the vehicle; Indicates the relative velocity relationship; Indicates the relative heading difference; Indicates lane ownership or lateral encroachment status; Indicates the occupancy status of potential conflict zones; It indicates the time to collision, the time to near-collision, or other indicators of collision intensity.

[0047] Calculate the planning relevance score based on shared scenario tokens and risk context vectors: (8) In the formula, Indicates the first Each token scores the relevance of the vehicle's future plans to the overall plan. This represents the risk scoring mapping function; Indicates the first A shared scenario token; Indicates the first The risk context vector of each token.

[0048] The risk weights are obtained by normalizing the planning relevance scores: (9) In the formula, Indicates the first Risk weight of each token; Indicates the first Planning relevance score for each token; Indicates the first The planning relevance score of each token; N represents the number of tokens participating in the risk weight allocation.

[0049] Construct risk-enhanced tokens based on risk weights: (10) In the formula, Indicates the first A risk-enhancing token; Indicates the first A shared scenario token; Indicates the first Risk weight of each token; This represents the risk context encoding function; This represents the risk context vector.

[0050] The risk-sparse token set consists of all risk-enhancing tokens. (11) In the formula, This represents the current set of sparse tokens representing risks related to the planning task. Indicates the first A risk-enhancing token; This indicates the number of risk-sparse tokens.

[0051] Construct a vehicle planning query based on the vehicle's status and the shared scenario context: (12) (13) In the formula, This indicates the current vehicle planning query. This indicates the mapping function for the query; Represents the vehicle's state vector; Global context features representing shared sparse scene representations; Indicates the first A shared scenario token; N represents the number of sparse anchor points.

[0052] Based on vehicle planning queries and risk sparse token sets, perform vehicle-centric geometric interaction modeling: (14) (15) In the formula, Indicates the first Each risk-enhancing token has a weighted interaction with the autonomous vehicle planning process; Represents the query mapping matrix; Represents the key mapping matrix; Represents a value mapping matrix; This indicates a vehicle planning query; Indicates the first A risk enhancement token; d represents the attention feature dimension; This represents the geometric interaction at the center of the vehicle.

[0053] Through the above process, the model can highlight vehicles that cut in at close range, potential pedestrians crossing the road, traffic participants occupying conflict areas, and key road areas that are highly related to the vehicle's planned path, avoiding equal weighting of all objects in the scene.

[0054] 4: Planning decoding, risk scoring, and confidence estimation; The sparse risk representation and the vehicle-centric geometric interaction representation are input into the planning and decoding module to generate candidate planned trajectories for the vehicle, and risk scoring and confidence estimation are performed on the candidate trajectories. The input representation of the planning and decoding module is as follows: (16) In the formula, This represents the input characteristics of the planning and decoding module; This indicates a pooling or aggregation operation on a sparse set of risk tokens; Represents a risk-sparse token set; This represents the geometric interaction representation of the vehicle center. This represents the vehicle's state vector.

[0055] M candidate trajectories for autonomous vehicles are generated based on the planning and decoding module: (17) In the formula, This represents the set of candidate planned trajectories for the vehicle generated at the current moment; This represents the m-th candidate trajectory for the autonomous vehicle. This represents the number of candidate trajectories; Dec_π(·) represents the planning decoding function; This represents the input characteristics of the planning decoding module.

[0056] No. The candidate planning trajectories are represented as follows: (18) In the formula, Indicates the first Candidate planning trajectories; and This indicates that the m-th candidate trajectory is at a future time. Location; Indicates the first Candidate trajectories at future moments The heading angle; Indicates the first Candidate trajectories at future moments speed; Indicates the length of the planning time domain.

[0057] Risk scoring is performed on each candidate planning trajectory: (19) In the formula, Indicates the first Risk scoring for each candidate planning trajectory; This represents the risk scoring function for candidate trajectories. Indicates the first Candidate planning trajectories; Represents a risk-sparse token set; This represents the geometric interaction at the center of the vehicle.

[0058] Confidence estimation is performed for each candidate planning trajectory: (20) In the formula, Indicates the first The confidence level of each candidate planning trajectory; This represents the confidence estimation function for candidate trajectories; Indicates the first Candidate planning trajectories; This represents the input characteristics of the planning decoding module.

[0059] In one alternative implementation, the planning decoding module can also generate future movement patterns of surrounding traffic participants to assist in risk scoring of candidate planning trajectories. (twenty one) In the formula, This represents the set of future movement patterns of surrounding traffic participants; Indicates the j-th surrounding traffic participant in the first place. The future trajectory under each motion pattern; J represents the number of surrounding traffic participants; K represents the number of motion patterns for each traffic participant. These future motion patterns serve as auxiliary information for risk scoring and safety constraint re-scoring, and are not included in the final system output.

[0060] 5: End-to-end joint optimization and security constraint re-evaluation.

[0061] The training phase is used to enable the collaborative optimization of shared sparse scene representation construction, risk sparse modeling, vehicle-centric geometric interaction modeling, and planning decoding within a unified framework; the inference phase is used to select the target planning trajectory that meets the scene constraints from the candidate planning trajectories.

[0062] During the training phase, a joint loss function is constructed: (twenty two) In the formula, This represents the total training loss from end to end; This represents the planning trajectory fitting loss, used to constrain the deviation between the candidate planning trajectory and the reference trajectory; This represents the risk scoring supervision loss, used to enhance the model's sensitivity to high-risk targets and conflict zones; This represents the supervised loss based on the confidence level of the candidate trajectory, used to constrain the reliability estimation of the candidate planned trajectory; This represents the safety constraint loss, used to constrain the trajectory to meet collision safety, road boundary, and smoothness requirements; , and These represent the weighting coefficients of the corresponding loss terms.

[0063] During the inference phase, a safety constraint re-evaluation cost is constructed for each candidate planning trajectory generated in step 4, expressed by the formula: (twenty three) In the formula, Indicates the first The comprehensive rescoring cost of candidate planning trajectories; Indicates the first Candidate planning trajectories; Indicates the risk and cost; Indicates the cost of collision; Indicates the cost at the road boundary; Indicates the cost of smoothing; Indicates the cost of dynamic executability; , , , and These represent the weight coefficients of the corresponding cost terms.

[0064] Finally, by minimizing the overall re-scoring cost, the target planning trajectory is selected from the set of candidate planning trajectories. (twenty four) In the formula, This represents the final output target trajectory; Represents the set of candidate planning trajectories; Represents the first in the candidate planning trajectory set. Candidate planning trajectories; This represents the overall rescoring cost corresponding to the candidate trajectory.

[0065] In summary, the multimodal interactive end-to-end autonomous driving method provided by this invention constructs a multimodal sparse scene representation for autonomous driving tasks, achieving collaborative modeling of image semantic information, point cloud geometric information, and vehicle state information within a unified feature space. Furthermore, by combining risk sparse representation with a vehicle-centric geometric interaction mechanism, it can more accurately identify targets and regions that have a critical impact on the vehicle's future trajectory. Through unified motion planning decoding and joint optimization of safety constraints, it improves the safety, stability, and executability of trajectory generation in complex traffic scenarios. Therefore, this invention can effectively enhance the environmental understanding and planning decision-making capabilities of end-to-end autonomous driving systems in lane changing, merging, intersection yielding, and dense traffic flow scenarios, possessing significant engineering application value and promotional value.

[0066] On the other hand, embodiments of the present invention propose a multimodal interactive end-to-end autonomous driving system, the system comprising: The multimodal input module is used to receive multi-channel surround view image data, LiDAR point cloud data, and vehicle status data at the current moment. The shared sparse scene representation construction module is used to set sparse anchor points in the vehicle coordinate system, and to perform anchor point-level alignment sampling and fusion of image semantic features and point cloud geometric features to form a unified shared sparse scene. The risk sparse representation and vehicle-centric geometric interaction module is used to extract the position, speed, heading, lane affiliation and potential conflict relationship of surrounding targets relative to the vehicle, and to perform risk weighting on the shared sparse scene. At the same time, it establishes geometric interaction relationship between the vehicle and key targets with the vehicle planning query as the center. The unified motion planning decoding and safety constraint screening module is used to simultaneously output the future trajectory patterns of surrounding targets and candidate planned trajectories of the vehicle based on shared interactive features, and to perform re-scoring and screening of candidate trajectories in combination with collision constraints, boundary constraints, smoothness constraints and dynamic constraints. The joint loss and phased training module is used to perform end-to-end optimization of the entire network using a multi-task loss function and a two-stage training strategy.

[0067] Figure 2The diagram shows a shared sparse scene representation construction module. In the diagram, the system first receives multi-channel surround view image data and LiDAR point cloud data, and extracts image semantic features and point cloud geometric features through image feature encoding networks and point cloud feature encoding networks, respectively. Then, a sparse scene anchor point set is constructed in a unified coordinate system, and anchor point projection and alignment sampling are performed on image features based on the sparse anchor points, while anchor point neighborhood geometric aggregation is performed on point cloud features. Finally, the image semantic response and point cloud geometric response are fused through a gated fusion unit to obtain an anchor point-aligned multimodal shared sparse scene token, providing input for subsequent risk sparse representation generation and vehicle-centric geometric interaction modeling.

[0068] Figure 3 The diagram illustrates the risk sparse representation and vehicle-centric geometric interaction module, along with the unified motion planning decoding and safety constraint screening module. In this module, the system uses a shared sparse scene token set as input and combines it with the vehicle's state sequence, historical trajectory information, and global scene context to construct a vehicle planning query vector. Subsequently, risk context information is generated based on the relative position, speed, heading, lane affiliation, and potential conflict relationships of surrounding traffic participants and key road areas relative to the vehicle, forming a risk enhancement token set. Based on this, the system performs geometric interaction modeling centered on the vehicle, obtaining a vehicle-centric interaction representation. Finally, the risk enhancement tokens and the vehicle-centric interaction representation are input into the planning decoding module to generate candidate vehicle planning trajectories. These candidate trajectories are then re-scored by the safety constraint scoring unit, outputting the final safe target planning trajectory.

[0069] In summary, this invention provides a multimodal interactive end-to-end autonomous driving system that uses multi-path surround view images, LiDAR point clouds, and vehicle state sequences as inputs. Through a unified sparse scene intermediate representation, it achieves synchronous modeling of the motion information of surrounding traffic participants and the future planned trajectory of the vehicle, and outputs target planning results that meet the requirements of safety, smoothness, and executability.

[0070] This invention also provides an electronic device comprising: a processor, a communications interface, a memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions in the memory to execute implementation steps of a multimodal interactive end-to-end autonomous driving method.

[0071] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0072] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0073] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0074] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal interactive end-to-end autonomous driving method, characterized in that, Includes the following steps: Acquire multimodal perception data and vehicle status data; the multimodal perception data includes multi-path surround view image data and lidar point cloud data; Feature encoding is performed on the obtained image data and point cloud data to extract semantic features from the image modality and geometric features from the point cloud modality. An anchor-aligned multimodal shared sparse scene representation is constructed in a unified coordinate system. Based on the obtained shared sparse scene representation, the spatiotemporal relationship information of surrounding traffic participants and key road areas relative to the vehicle is extracted, and risk sparse representation and vehicle-centric geometric interaction model are established for planning tasks. The obtained sparse risk representation and the vehicle-centric geometric interaction representation are used to plan and decode to generate candidate planning trajectories for the vehicle, and risk scoring and confidence estimation are performed on the candidate planning trajectories. During the training phase, the construction of shared sparse scene representations, risk sparse modeling, vehicle-centric geometric interaction modeling, and planning decoding are incorporated into a unified end-to-end training framework for joint optimization. During the inference phase, the candidate planning trajectories are re-scored based on the jointly optimized model to meet safety constraints, and the target planning trajectory that satisfies the scenario constraints is output.

2. The multimodal interactive end-to-end autonomous driving method according to claim 1, characterized in that, The vehicle status data includes one or more of the following: vehicle position and attitude, speed, acceleration, heading angle, yaw rate, steering status, and historical trajectory information.

3. The multimodal interactive end-to-end autonomous driving method according to claim 1, characterized in that, After acquiring the multimodal perception data and the vehicle status data, the process also includes: The image data is scaled and brightness standardized, including: denoising, outlier removal and coordinate transformation of point cloud data; time alignment and sequence caching of vehicle status data, and mapping of multimodal inputs to the vehicle coordinate system or bird's-eye view coordinate system.

4. The multimodal interactive end-to-end autonomous driving method according to claim 1, characterized in that, The step of performing feature encoding on the obtained image data and point cloud data to extract semantic features from the image modality and geometric features from the point cloud modality includes: Feature encoding is performed on the obtained image data and point cloud data respectively to extract semantic features from the image modality and geometric features from the point cloud modality; A set of sparse scene anchor points is constructed under a unified coordinate system. The sparse scene anchor points are used to characterize key locations, candidate traffic participants and potential conflict areas in the road space. Project, sample, and aggregate image semantic features and point cloud geometric features onto sparse scene anchor points; By sampling and fusing cross-modal features, a unified shared sparse representation unit is formed at each sparse anchor point, so that the shared sparse representation unit contains both semantic and geometric information.

5. The multimodal interactive end-to-end autonomous driving method according to claim 1, characterized in that, Based on the obtained shared sparse scene representation, the spatiotemporal relationship information of surrounding traffic participants and key road areas relative to the vehicle is extracted, and risk sparse representation and vehicle-centric geometric interaction modeling for planning tasks are established, including: Based on the obtained shared sparse scene representation, the spatiotemporal relationship information of surrounding traffic participants and key road areas relative to the vehicle is extracted, including relative position, relative speed, relative heading, lane ownership relationship, lateral encroachment relationship, potential conflict area occupancy status and historical movement change trend. Assess the impact of each traffic participant and spatial area on the future trajectory of the vehicle, and assign corresponding risk weights to each traffic participant and spatial area according to the impact level, so that targets and areas with stronger correlation to the future trajectory of the vehicle and higher potential conflict level can obtain stronger feature responses in the risk sparse representation, so as to form a risk-enhanced sparse representation for planning tasks. Using the current state of the vehicle and the planning query as the interaction center, the geometric relationship between the key targets in the risk-enhanced sparse representation and the vehicle is encoded to construct a vehicle-centric geometric interaction representation; By jointly modeling risk weights and geometric interactions, priority is given to key targets and key areas that have a real impact on the future trajectory of the vehicle, rather than treating all objects in the scene equally.

6. The multimodal interactive end-to-end autonomous driving method according to claim 5, characterized in that, The risk weights assigned to each traffic participant and spatial area based on the degree of impact include: The targets and areas are divided into high-risk, medium-risk, and low-risk targets. High-risk targets are vehicles that cut in at close range and potential pedestrians crossing the road. Medium-risk targets are nearby vehicles or dynamic targets to be observed that may intersect with the vehicle's trajectory. Low-risk targets are distant targets or non-critical static areas that are not strongly associated with the vehicle's movement path.

7. The multimodal interactive end-to-end autonomous driving method according to claim 1, characterized in that, The process of planning and decoding the obtained sparse risk representation and the vehicle-centric geometric interaction representation includes: Based on the vehicle's current motion state, the risk weights and interaction relationships of surrounding key targets, the candidate trajectory or target trajectory of the vehicle in the future time domain is output. During the trajectory generation process, the risk score, confidence score, or distribution parameters of the candidate trajectories are output to characterize the safety and feasibility of different planning results in the current traffic scenario. Prioritize retaining low-risk, highly feasible candidate planning results, suppress high-risk candidate trajectories that may lead to collisions, boundary violations, or significant oscillations, and provide the screened candidate planning trajectories to the safety constraint re-evaluation process.

8. The multimodal interactive end-to-end autonomous driving method according to claim 1, characterized in that, The joint optimization in the unified end-to-end training framework includes: Construct a joint loss function that includes planning fitting loss and safety constraint loss. The planning fitting loss is used to constrain the positional deviation, orientation deviation or dynamic deviation between the predicted trajectory and the reference trajectory, and the safety constraint loss is used to constrain the planning result to meet the real driving safety requirements. Introduce risk-leading terms or interactive salience-leading terms to enhance sensitivity to risk objectives and conflict areas; After training, an end-to-end autonomous driving model is obtained. The autonomous driving model receives multimodal environment input and outputs the future trajectory of the vehicle that meets the requirements of safety, stability and executability.

9. A multimodal interactive end-to-end autonomous driving system, characterized in that, include: The multimodal input module is used to receive multi-channel surround view image data, LiDAR point cloud data, and vehicle status data at the current moment. The shared sparse scene representation construction module is used to set sparse anchor points in the vehicle coordinate system, and to perform anchor point-level alignment sampling and fusion of image semantic features and point cloud geometric features to form a unified shared sparse scene. The risk sparse representation and vehicle-centric geometric interaction module is used to extract the position, speed, heading, lane affiliation and potential conflict relationship of surrounding targets relative to the vehicle, and to perform risk weighting on the shared sparse scene. At the same time, it establishes geometric interaction relationship between the vehicle and key targets with the vehicle planning query as the center. The unified motion planning decoding and safety constraint screening module is used to simultaneously output the future trajectory patterns of surrounding targets and candidate planned trajectories of the vehicle based on shared interactive features, and to perform re-scoring and screening of candidate trajectories in combination with collision constraints, boundary constraints, smoothness constraints and dynamic constraints. The joint loss and phased training module is used to perform end-to-end optimization of the entire network using a multi-task loss function and a phased training strategy.

10. An electronic device, characterized in that, It includes a memory and a processor, wherein the processor is used to implement a multimodal interactive end-to-end autonomous driving method as described in any one of claims 1 to 8 when executing a computer program stored in the memory.

Images