Autonomous driving decision-making method based on three-dimensional scene reconstruction and multi-modal large model

Abstract

The application relates to the technical field of automatic driving, in particular to an automatic driving decision method based on three-dimensional scene reconstruction and a multi-modal large model, which comprises the following steps: incrementally modeling a static background at a current time step in a driving scene as a static 3D Gaussian model to obtain a background Gaussian model; constructing a compound dynamic Gaussian graph of dynamic objects at the current time step in the driving scene to obtain a dynamic object Gaussian graph; integrating the dynamic object Gaussian graph into the background Gaussian model to obtain a road condition Gaussian field at the current time step; rendering the road condition Gaussian field according to multi-view images to obtain a road condition planar image at the current time step; and obtaining a driving path at a next time step based on the real-time road condition planar image and a trained automatic driving network system. When the driving path is predicted according to the road condition planar image, the multi-modal large model LLM can be combined to realize intention understanding and planning alignment.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an autonomous driving decision-making method based on 3D scene reconstruction and multimodal large models. Background Technology

[0002] The intelligent transportation multimodal perception and decision-making system overcomes the limitations of traditional SLAM and NeRF in dynamic scenes, providing robust environmental modeling capabilities for Level 4 autonomous driving. Addressing the challenges of frequent changes in static backgrounds and the variability of dynamic objects in complex dynamic scenes, we propose an incremental 3D Gaussian modeling and dynamic reconstruction method. By combining incremental learning with Gaussian representation, balancing efficiency and accuracy, and adapting to onboard computing power limitations, we significantly reduce resource consumption and costs. Furthermore, we support multi-scale scene reconstruction from open roads to closed environments (such as mines and warehouses), covering the needs of Level 2-4 autonomous driving.

[0003] Vehicles encounter large-scale, long-lasting static backgrounds during operation, as well as frequently changing objects within those backgrounds as the vehicle moves. Examples include urban road construction and temporary obstacles suddenly appearing in the field of view. This makes existing modeling frameworks prone to artifacts and blurring, leading to perceptual distortion. Our proposed incremental static 3D Gaussian model significantly improves reconstruction accuracy by updating background geometry in real time. It not only greatly reduces lane line continuity errors, such as in accurate modeling of lane lines and buildings in urban NOA (Navigate on Autopilot) systems, but also provides a reliable foundation for high-precision positioning and path planning.

[0004] In urban autonomous driving, real-time reconstruction of suddenly appearing vehicles is particularly important, especially in unstructured scenarios. However, existing 3D-GS (3D-Gaussia Splatting) methods struggle to characterize the long-term motion of multiple targets. For instance, in driving scenarios with dense traffic, pedestrian trajectories are difficult to predict. The integrated dynamic Gaussian map proposed by the intelligent transportation multimodal perception and decision-making system achieves accurate geometric reconstruction and trajectory prediction of dynamic objects through spatiotemporal joint modeling, significantly reducing tracking errors to within 0.5m and solving the problems of missed detections and false detections.

[0005] To reduce the cost of 3D scene reconstruction for autonomous driving systems, this application proposes an autonomous driving decision-making method based on 3D scene reconstruction and a multimodal large model, which introduces a large language model to assist the autonomous driving system in making decisions. Summary of the Invention

[0006] To overcome the problems existing in related technologies, this application provides an autonomous driving decision-making method based on 3D scene reconstruction and multimodal large model, including the following steps:

[0007] Incremental static 3D Gaussian modeling is performed on the static background of the current time step in the driving scene to obtain the background Gaussian model; the parameters of the background Gaussian model of the current time step are updated using the background Gaussian model of the previous time step as a priori.

[0008] A composite dynamic Gaussian graph is constructed for the dynamic objects at the current time step in the driving scene to obtain the dynamic object Gaussian graph;

[0009] The dynamic object Gaussian map is integrated into the background Gaussian model to obtain the road condition Gaussian field at the current time step;

[0010] The road condition Gaussian field is rendered based on the multi-view images to obtain the road condition planar image at the current time step;

[0011] The driving path for the next time step is obtained based on the real-time road condition plan image and the trained autonomous driving network system; the autonomous driving network system is trained through an agent-centered learning paradigm module and an autonomous vehicle-centered learning paradigm module.

[0012] In one implementation, the step of performing incremental static 3D Gaussian modeling on the static background at the current time step in the driving scene to obtain a background Gaussian model specifically includes:

[0013] Obtain the background point cloud data, background image, and background Gaussian model from the previous time step at the current time step;

[0014] The background Gaussian model of the previous time step is used as a priori to determine the parameters of the background Gaussian model of the current time step; the parameters of the background Gaussian model include: position, covariance matrix, spherical harmonic function coefficients, and opacity;

[0015] The parameters of the background Gaussian model are updated using the background point cloud data and the background image at the current time step to obtain the background Gaussian model at the current time step.

[0016] In one implementation, the use of the background Gaussian model from the previous time step as parameters for determining the background Gaussian model at the current time step specifically includes:

[0017] Align the overlapping regions of the background Gaussian model from the previous time step and the background Gaussian model from the current time step, merge the Gaussian coordinates of the overlapping regions into the background Gaussian model from the previous time step, and initialize the background Gaussian model from the current time step.

[0018] In one implementation, updating the parameters of the background Gaussian model using the background point cloud data and the background image at the current time step specifically includes:

[0019] Update the position and covariance matrix of the non-overlapping regions in the background Gaussian model at the current time step using the background point cloud data at the current time step;

[0020] The spherical harmonic coefficients and opacity in the background Gaussian model at the current time step are updated with the background image.

[0021] In one implementation, integrating the dynamic object Gaussian map into the background Gaussian model to obtain the road condition Gaussian field at the current time step specifically includes:

[0022] The opacity of the Gaussian map of the dynamic object is determined based on the distance between the Gaussian map of the dynamic object and the coordinates of the camera center.

[0023] Using the adjusted opacity as the center of the Gaussian distribution of the dynamic object, calculate the transformation matrix of the Gaussian distribution of the dynamic object to the background Gaussian model.

[0024] The Gaussian map of the dynamic object is stitched into the background Gaussian model in chronological order using a transformation matrix to obtain the Gaussian field of the road condition.

[0025] In one implementation, rendering the road condition Gaussian field based on multi-view images to obtain the dynamic driving scene at the current time step specifically includes:

[0026] The road condition Gaussian field is mapped onto a two-dimensional plane using a 3D Gaussian splash renderer to obtain the road condition plane image.

[0027] In one implementation, obtaining the driving path for the next time step based on the real-time road condition planar image and the trained autonomous driving network system specifically includes:

[0028] The road condition planar image is annotated and segmented to obtain the BEV feature tensor;

[0029] Determine the expected feature tensor based on the BEV feature tensor;

[0030] The travel path for the next time step is determined based on the desired feature tensor.

[0031] In one implementation, the road condition planar image is annotated and segmented to obtain a BEV feature tensor, specifically including:

[0032] The vehicle and foreground object regions are clipped using a 3D bounding box;

[0033] The lane area is segmented using a general visual scene mask;

[0034] Pooling is performed on the clipped regions to generate feature representations for each agent, and then the representations are concatenated along the batch direction to form the agent BEV feature tensor.

[0035] In one implementation, determining the desired feature tensor based on the BEV feature tensor specifically includes:

[0036] The multilayer perceptron is adapted to the BEV space and connected to the proxy expected feature tensor.

[0037] In one implementation, the method of determining the travel path for the next time step based on the desired feature tensor specifically includes:

[0038] Extract self-query features based on the expected feature tensor; determine the driving path for the next time step based on the self-query features.

[0039] The technical solution provided in this application may include the following beneficial effects:

[0040] This application constructs a background Gaussian model through incremental static 3D Gaussian modeling, integrates the Gaussian map of dynamic objects into the background Gaussian model to obtain a road condition Gaussian field at multiple time steps, and renders the road condition Gaussian field to obtain a bird's-eye view of the road condition planar image. The autonomous driving system is trained using both an agent-centric learning paradigm module and a vehicle-centric learning paradigm module. When predicting driving paths based on the road condition planar image, it can combine a multimodal large model (LLM) to achieve alignment between intent understanding and planning.

[0041] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0042] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0043] Figure 1 This is a flowchart illustrating the autonomous driving decision-making method based on 3D scene reconstruction and multimodal large model as shown in the embodiments of this application. Detailed Implementation

[0044] Preferred embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0045] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0046] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0047] To address the problems in the existing technology, this application provides an autonomous driving decision-making method based on 3D scene reconstruction and multimodal large model.

[0048] This autonomous driving decision-making method is based on an autonomous driving decision-making system, which includes: a static background modeling module, responsible for constructing and updating a 3D Gaussian model of the static background in the driving scene, providing a stable environmental representation for the system; a dynamic object modeling module, focusing on detecting, segmenting, and modeling dynamic objects, generating dynamic Gaussian maps to capture their spatiotemporal motion characteristics; a scene integration module, which merges the static background and dynamic object models to generate a unified scene representation, providing high-quality input data for subsequent decisions; and a vision-language-planning module, which enhances scene understanding and decision planning by combining the reasoning capabilities of a language model with a vision-language-planning framework.

[0049] Figure 1 This is a flowchart illustrating the autonomous driving decision-making method based on 3D scene reconstruction and multimodal large model as shown in the embodiments of this application.

[0050] like Figure 1 As shown in the embodiments of this application, the autonomous driving decision-making method includes the following steps:

[0051] S1. Perform incremental static 3D Gaussian modeling on the static background of the current time step in the driving scene to obtain the background Gaussian model; update the parameters of the background Gaussian model of the current time step using the background Gaussian model of the previous time step as a priori.

[0052] Furthermore, step S1 specifically includes:

[0053] S101. Obtain the background point cloud data, background image, and background Gaussian model of the previous time step at the current time step;

[0054] S102. The background Gaussian model of the previous time step is used as a priori to determine the parameters of the background Gaussian model of the current time step; the parameters of the background Gaussian model include: position, covariance matrix, spherical harmonic function coefficients, and opacity.

[0055] S103. Update the parameters of the background Gaussian model with the background point cloud data and the background image at the current time step to obtain the background Gaussian model at the current time step.

[0056] In step S102, the overlapping regions of the background Gaussian model in the previous time step and the background Gaussian model in the current time step are aligned, the Gaussian coordinates of the overlapping regions are merged into the background Gaussian model in the previous time step, and the background Gaussian model in the current time step is initialized.

[0057] Specifically, in the first time step, the background point cloud data is used as a priori to initialize the Gaussian model.

[0058] Specifically, the location distribution of radar points is described using a non-normalized Gaussian function. Indicates the prior position of the lidar;

[0059]

[0060] in, It is a non-normalized Gaussian function used to describe the location distribution of radar points; It is the average value of the LiDAR points; Let covariance matrix be the variance matrix. ; Three-dimensional coordinates of the radar point

[0061] Through orthogonal rotation matrix Translation vector The 3D coordinates of the background Gaussian image are transformed to the camera coordinate system, the 2D pixel coordinates of these 3D coordinates are determined, and then matched to the background image through projection. 2D pixels on a plane Among the LiDAR points projected onto multiple pixels in multiple images, the point closest to the image plane is selected as the projection point and assigned a color. For example, the mathematical representation of color assignment for the background Gaussian model is as follows:

[0062]

[0063] in, These are the two-dimensional pixel coordinates of the LiDAR points projected onto the camera image. The horizontal pixel coordinates of the image. These are the vertical pixel coordinates of the image. The coordinates of a three-dimensional point measured by lidar. It is an orthogonal rotation matrix. It is a translation vector. This is the camera intrinsic parameter matrix.

[0064] This application embodiment uses incremental static 3D Gaussian modeling to construct a 3D scene. Based on the depth range provided by the LiDAR prior, the static background is divided into N intervals arranged in a time series. Furthermore, the background Gaussian model for subsequent time steps is determined by utilizing the perspective changes caused by vehicle movement and the temporal relationship between adjacent frames.

[0065] Step S103 specifically includes:

[0066] S1031. Update the position and covariance matrix of the non-overlapping region in the background Gaussian model at the current time step using the background point cloud data at the current time step;

[0067] S1032. Update the spherical harmonic function coefficients and opacity in the background Gaussian model at the current time step with the background image.

[0068] Specifically, in each subsequent time step In this process, the surrounding view at each time step is used as supervision to update the parameters of the Gaussian model, including location. Covariance matrix spherical harmonic coefficients and opacity Using the background Gaussian model from the previous time step as the position prior for the next time step, and aligning adjacent intervals based on overlapping regions, the set of 3D center points of the Gaussian model in the overlapping regions is obtained. Gaussian coordinates within the non-overlapping region Combined, for example, the mathematical representation of combined Gaussian coordinates is:

[0069]

[0070] in, For the first A set of Gaussian coordinates for each time step. The set of Gaussian coordinates for the overlapping region. These are the Gaussian coordinates for the non-overlapping region.

[0071] The color and opacity of the background Gaussian model are then determined iteratively.

[0072] The scene within subsequent intervals is integrated into the previously constructed background Gaussian model, with multiple surrounding frames used as supervision. This is based on the color corresponding to a single Gaussian model from a given viewpoint. Opacity By calculating all intervals Obtain the cumulative transmittance of the scene. The final background Gaussian model is obtained. .

[0073] In this process, the implicit alignment of the Gaussian model is formed by relying on the overlapping areas between the surrounding multi-camera images.

[0074]

[0075] in, Incremental static 3D Gaussian model The final color.

[0076] During the 3D Gaussian projection process in the reconstruction, a weighted averaging method is used to reconstruct the scene's colors as accurately as possible, reducing potential differences in sampling of the same scene by front and rear cameras. The optimized pixel colors are then utilized. Differential sputtering, weights from different perspectives and a view matrix for aligning multi-camera views. For example, the color reconstruction process is represented as follows:

[0077]

[0078] S2. Construct a composite dynamic Gaussian graph for the dynamic objects at the current time step in the driving scene to obtain the dynamic object Gaussian graph.

[0079] Step S2 includes the following steps:

[0080] S201. First, using the bounding boxes provided by the dataset, dynamic foreground objects are separated from the static background, and a dynamic Gaussian map is constructed. Grounded SAM is then used to perform accurate pixel-level dynamic object extraction based on the bounding box range. The extracted dynamic objects are identified by their object ID and the timestamp of their appearance.

[0081] S202, Each node stores one instance object. To represent the corresponding dynamic Gaussian model, and the transformation matrix for each object. And based on the center coordinates of the bounding box. and time step The direction of the time bounding box This allows us to construct a dynamic Gaussian plot and calculate a Gaussian model for each dynamic object individually.

[0082]

[0083] After obtaining the background Gaussian model and the dynamic object Gaussian map, the dynamic object Gaussian map is added to the background Gaussian model, as shown in step S3. Step S3 includes:

[0084] S301. Determine the opacity of the Gaussian image of the dynamic object based on the distance between the Gaussian image of the dynamic object and the coordinates of the camera center.

[0085] S302. Using the adjusted opacity as the center of the Gaussian distribution of the dynamic object, calculate the transformation matrix of the Gaussian distribution of the dynamic object to the background Gaussian model.

[0086] S303. By using a transformation matrix, the Gaussian map of the dynamic object is stitched into the background Gaussian model in chronological order to obtain the Gaussian field of the road condition.

[0087] Specifically, in step S302, the transformation matrix and the rotation matrix of each dynamic object are used. Translation matrix To target dynamic objects The coordinate system is transformed to the global coordinate system of the static background. For example:

[0088]

[0089] in, Let be the transformation matrix of the Gaussian graph of the current dynamic object. Let be the rotation matrix of the Gaussian graph of the current dynamic object. This is the translation matrix of the Gaussian graph of the current dynamic object.

[0090] Specifically, in step S303, the Gaussian distribution of the nodes is stitched into the background Gaussian model in chronological order according to the position and orientation of the bounding box, and based on the relationship with the camera center... The opacity is adjusted based on the distance; the initial Gaussian model opacity is... The time step is obtained by adjusting the opacity. Time object Opacity And using the object's Gaussian center, calculate the transformation matrix from the dynamic object's Gaussian map to the background Gaussian model. .

[0091] For example, the formula for calculating the transformation matrix is:

[0092]

[0093] in, The center of Gauss of the object, This represents the center position of the bounding box of object o.

[0094] Finally, the optimized dynamic Gaussian graph H is compared with the static Gaussian model. Merging the results yields the road condition Gaussian field. The road condition Gaussian field is as follows:

[0095]

[0096] S4. Render the Gaussian field of the road condition based on the multi-view images to obtain the road condition planar image at the current time step;

[0097] Step S4 specifically includes: mapping the road condition Gaussian field onto a two-dimensional plane using a 3D Gaussian splash renderer to obtain the road condition plane image.

[0098] Specifically, including:

[0099] 1. Use a differentiable 3D Gaussian splash renderer ς Map the globally integrated 3D Gaussian model onto a 2D plane. This is achieved using the Jacobian matrix in the perspective view. , and represent the transformation matrix from the world coordinate system to the camera coordinate system. To calculate the covariance matrix: .

[0100] 2. The road condition Gaussian field projects the global 3D Gaussian model onto multiple 2D planes and provides an observation of the surrounding objects at each step.

[0101] During global rendering, subsequent Gaussian models are initially not visible from the current representation, but are subsequently incorporated into the model under supervision that matches the global image.

[0102] The embodiments of this application use a loss function consisting of three parts.

[0103] First, TSSIM (Tile Structure Similarity) is integrated into Gaussian sputtering to evaluate the similarity between rendered tiles and their respective ground truth values. Furthermore, in this equation, the screen is divided into... A tile, These represent the training parameters of the Gaussian model. Represents the rendered tile generated by Gaussian sputtering, and specifies the ground truth value of the corresponding real tile. The loss term based on tile structure similarity is expressed as:

[0104]

[0105] in, The loss term is based on tile structure similarity. This is a structural similarity index.

[0106] Second, a robust loss mechanism is designed to reduce outliers in the Gaussian field of the road condition, using shape parameters. To adjust the sustainability of losses, and Representing real and synthetic images respectively, the shape loss term is:

[0107]

[0108] in, It is a component of the loss function, used to reduce outliers in a 3D Gaussian distribution.

[0109] Third, use lidar loss control to predict the location of the Gaussian field of road conditions. To enhance the geometry and edge patterns, and through Point Prior The loss term for lidar is:

[0110]

[0111] Ultimately, the road condition Gaussian field was optimized by minimizing the sum of these three loss components.

[0112] BEV (Browser-Electronic Vehicle) is based on images taken by cameras from different perspectives, rather than on a true bird's-eye view. This can lead to deviations from the ideal BEV and affect the safety and accuracy of the decoder's conclusions. To minimize this impact, the autonomous driving system proposed in this application uses an agent-centered learning paradigm that integrates an integrated language encoder (LM) for comparative learning. The agent-centered learning paradigm method uses the BEV feature tensor generated through comparative learning... With expected feature tensor Matching and using LLM to directly analyze BEV maps to reveal details and match them with human perception.

[0113] S5. Based on the real-time road condition planar image and the trained autonomous driving network system, the driving path for the next time step is obtained.

[0114] Specifically, the autonomous driving network system is trained using a learning paradigm module centered on intelligent agents and a learning paradigm module centered on autonomous vehicles.

[0115] After determining the road condition planar image, this embodiment of the application models and predicts trajectories for multiple agents in the road condition planar image, fusing ensemble language encoders, contrastive learning, and cross-entropy loss. The learning paradigm method adopted in this application uses the BEV feature tensor generated through comparative learning... With expected feature tensor Matching and using LLM to directly analyze BEV maps to reveal details and match them with human perception.

[0116] Step S5 specifically includes:

[0117] S501. Perform region annotation and segmentation on the road condition planar image to obtain the BEV feature tensor;

[0118] S502. Determine the expected feature tensor based on the BEV feature tensor;

[0119] S503. Determine the driving path for the next time step based on the desired feature tensor.

[0120] Step S501 specifically includes:

[0121] S5011, The vehicle and foreground object regions are clipped using a 3D bounding box;

[0122] S5012, the lane area is segmented using a general visual scene mask;

[0123] S5013. Pool the clipped regions to generate feature representations for each agent, and then concatenate them along the batch direction to form the agent BEV feature tensor.

[0124] In this embodiment of the application, each agent The annotation area is aligned with and cropped, and the areas of the vehicle and foreground objects are bounded by a 3D bounding box. The lane area is cropped using a general visual scene mask. Segmentation. Pool the cropped regions to generate feature representations for each agent, then concatenate them along the batch direction. Forming the proxy BEV feature tensor .

[0125]

[0126]

[0127]

[0128]

[0129] In step S502, a combination of language modeling and contrastive learning is used to ensure that the BEV features accurately reflect the expected information. Information about the expected agent, such as classification labels, frame limits, and trajectories, is formatted as text prompts and submitted to the LLM to generate the expected feature tensor. These representations are then adapted to the BEV space using a multilayer perceptron (MLP).

[0130] During training, the parameters of the language model are frozen to adapt to training that maintains a certain level.

[0131] The mathematical representation of the expected feature tensor is:

[0132]

[0133]

[0134]

[0135]

[0136] in, Let the expected feature tensor of the vehicle be... For the prospect expectation feature tensor, For the lane expectation feature tensor, For the agent's expected feature map; This is a text template for the vehicle's status. A text template to describe other traffic participants in the foreground. A text template for describing lane lines or road topology; For filling in The actual value of the vehicle's status is represented by the placeholder in the middle. Foreground participants are accurately labeled, including their category, location, speed, etc. The actual geometric annotation of lane lines is usually a series of coordinate points or curve parameters. This is the activation function.

[0137] A learning paradigm module centered on the agent is constructed, which aligns the agent's BEV feature tensor and agent's expected feature map through comparative learning after obtaining them.

[0138] This learning paradigm module includes L2 normalization, matrix multiplication, and logarithmic scaling training to generate predicted similarity matrices and compare them with identity matrices. Along of Modal axis Calculate the cross-entropy loss for each modal axis. The loss is then averaged to obtain the final agent-centric learning paradigm loss for optimizing BEV extrapolation. .

[0139]

[0140]

[0141] in, This is the similarity score matrix between the two sets of vectors predicted by the model. This is the ideal label matrix, i.e., an identity matrix, used to guide contrastive learning.

[0142] The construction of a learning paradigm module centered on autonomous vehicles can be summarized as follows: In the initial stage, the self-queries and interacts with other agents on the bird's-eye view (BEV) of lane information to extract perception and prediction data, and then uses the intermediate module of the information integration and planning team to predict future routes.

[0143] Furthermore, step S503 specifically includes:

[0144] S5031. Extract self-query features based on the expected feature tensor;

[0145] S5032. Determine the driving path for the next time step based on the self-query features.

[0146] Self-modeling of vehicle dynamics is a crucial component of autonomous driving systems. Traditional Adaptive Dash Modules (ADS) rely on trainable self-queries interacting with foreground and lane data in BEV maps to gather perception and prediction information and determine future driving paths. However, this approach may suffer from biased feature learning due to its over-reliance on numerical data, thus limiting the system's adaptability to new environments.

[0147] In step S5031, the self-query is compared with the foreground query on the BEV map. Lane Inquiry Communication is key. Through this interaction, the system can collect the perception information needed for autonomous driving, such as the state and predictive information of the surrounding environment, including the movement trends of other objects. By interacting with the BEV map, the system generates self-defined vehicle query features, which are the vehicle's understanding of its own state and environment. These features are further processed to predict the autonomous vehicle's driving path.

[0148] In simple terms, the system calculates the vehicle's next move based on current perception and prediction data. For example, it generates a planned path point (denoted as P×2) containing several future time steps (P). Here, P is the number of time steps, and 2 represents the two-dimensional coordinates of each time step. In this process, the batch size B during training determines the amount of data the system processes at one time.

[0149]

[0150]

[0151] in, and These represent the intermediate self in the formation assembly module and the planning head in ADS, respectively. For lane query features, Foreground query features, This is the trajectory for the next time step.

[0152] The technical solution proposed in this application, with its learning paradigm centered on autonomous vehicles, implements a language description of the vehicle's self-state through Ground Truth Planning (GT) templates. The method includes advanced driving instructions and driving trajectories. These descriptions are used as cues for the language model (LM) and, through the obtained ground truth values, are then used for planning. This enables the generation of realistic planning functions that integrate driving position information and human driving logic.

[0153] Then, the multilayer perceptron (MLP) adjusts the text planning features based on the search query feature space.

[0154]

[0155] in, This provides a self-description prompt for use in a learning paradigm centered on autonomous driving. This represents the actual ground value. This is a function representation for the adaptation layer.

[0156] In this embodiment, the LM is a pre-prepared model that shares both an agent-centric learning paradigm and an autonomous vehicle-centric learning paradigm. During training, only the adaptation layer needs to be adjusted. Minimize memory usage while preserving the knowledge of the pre-trained LM.

[0157] In the learning module centered on autonomous vehicles, it is necessary to align the vehicle's self-query features and text planning features, and based on a learnable logarithmic scale. ,predict and reality Calculate using the similarity matrix:

[0158]

[0159]

[0160] Finally, we calculate the encoder inference loss and decoder decision loss during training using the loss function. Because in our visual language planning training, the total loss... Inference loss from bird's-eye view (BEV) encoder and decoder decision loss with their respective weights and Its composition, mathematically expressed as:

[0161]

[0162] In this embodiment of the application, the encoder inference loss Derived from the agent-centered learning paradigm module, it is directly defined as The decoder decision loss integrates the losses from all downstream tasks in the autonomous driving system (ADS), including perception, prediction, and planning, as well as the losses arising from the autonomous vehicle-centric learning paradigm. In this embodiment, the total loss effectively combines the contributions of the encoder and decoder, ensuring comprehensive optimization of the training process.

[0163] This application achieves equivalent perception performance with a low-cost camera solution through a pure vision-language reasoning-planning framework, significantly improving the average accuracy (mAP) of current detection methods. Furthermore, BEV (Bird's Eye View) feature maps serve as the source memory pool for downstream decoding tasks in autonomous driving; however, their spatial consistency and robustness are often affected by biases in multi-view camera image transformation. To address this, an agent-centric learning paradigm module can be introduced to improve the consistency of BEV maps by actively shaping semantic knowledge to align features of agents (Ego-car, Foreground Objects, Lane Elements) within the BEV map.

[0164] In the ADS (Autonomous Driving Systems) of this application, the agent-centered learning paradigm and the autonomous driving-centered learning paradigm are combined with a multimodal large model (LLM) to achieve alignment between intent understanding and planning.

[0165] The solution of this application has been described in detail above with reference to the accompanying drawings. In the above embodiments, the descriptions of each embodiment have different emphases; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art should also understand that the actions and modules involved in the specification are not necessarily essential to this application. Furthermore, it is understood that the steps in the method of this application embodiment can be adjusted, combined, and deleted according to actual needs, and the modules in the device of this application embodiment can be combined, divided, and deleted according to actual needs.

[0166] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0167] Alternatively, this application may be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) thereon, which, when executed by a processor of an electronic device (or electronic device, server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0168] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the present application can be implemented as electronic hardware, computer software, or a combination of both.

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0170] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An autonomous driving decision-making method based on 3D scene reconstruction and multimodal large model, characterized in that, Includes the following steps: Incremental static 3D Gaussian modeling is performed on the static background of the current time step in the driving scene to obtain the background Gaussian model; the parameters of the background Gaussian model of the current time step are updated using the background Gaussian model of the previous time step as a priori. A composite dynamic Gaussian graph is constructed for the dynamic objects at the current time step in the driving scene to obtain the dynamic object Gaussian graph; The dynamic object Gaussian map is integrated into the background Gaussian model to obtain the road condition Gaussian field at the current time step; The road condition Gaussian field is rendered based on the multi-view images to obtain the road condition planar image at the current time step; The driving path for the next time step is obtained based on the real-time road condition planar image and the trained autonomous driving network system; the autonomous driving network system is trained through an agent-centered learning paradigm module and an autonomous vehicle-centered learning paradigm module. Specifically, obtaining the driving path for the next time step based on the real-time road condition planar image and the trained autonomous driving network system includes: The road condition planar image is annotated and segmented to obtain the BEV feature tensor; Determine the expected feature tensor based on the BEV feature tensor; The travel path for the next time step is determined based on the desired feature tensor. Specifically, the BEV feature tensor is obtained by performing region annotation and segmentation on the road condition planar image, including: The vehicle and foreground object regions are clipped using a 3D bounding box; The lane area is segmented using a general visual scene mask; Pooling is performed on the clipped regions to generate feature representations for each agent, and then the representations are concatenated along the batch direction to form the agent BEV feature tensor. Specifically, determining the desired feature tensor based on the BEV feature tensor includes: The BEV space is adapted using a multilayer perceptron and connected to the agent's expected feature tensor; wherein, a combination of language modeling and contrastive learning is used to ensure that the BEV features accurately reflect the expected information; information about the expected agent is formatted as text prompts and submitted to a multimodal large model to generate the expected feature tensor; these representations are adapted to the BEV space using the multilayer perceptron; Specifically, determining the travel path for the next time step based on the desired feature tensor includes: Extract self-query features based on the expected feature tensor; The driving path for the next time step is determined based on the self-query features.

2. The autonomous driving decision-making method based on 3D scene reconstruction and multimodal large model according to claim 1, characterized in that, The incremental static 3D Gaussian modeling of the static background at the current time step in the driving scene to obtain the background Gaussian model specifically includes: Obtain the background point cloud data, background image, and background Gaussian model from the previous time step at the current time step; The background Gaussian model of the previous time step is used as a priori to determine the parameters of the background Gaussian model of the current time step; the parameters of the background Gaussian model include: position, covariance matrix, spherical harmonic function coefficients, and opacity; The parameters of the background Gaussian model are updated using the background point cloud data and the background image at the current time step to obtain the background Gaussian model at the current time step.

3. The autonomous driving decision-making method based on 3D scene reconstruction and multimodal large model according to claim 2, characterized in that, The background Gaussian model from the previous time step is used as a priori parameters to determine the background Gaussian model for the current time step, specifically including: Align the overlapping regions of the background Gaussian model from the previous time step and the background Gaussian model from the current time step, merge the Gaussian coordinates of the overlapping regions into the background Gaussian model from the previous time step, and initialize the background Gaussian model from the current time step.

4. The autonomous driving decision-making method based on 3D scene reconstruction and multimodal large model according to claim 3, characterized in that, The step of updating the parameters of the background Gaussian model using the background point cloud data and the background image at the current time step specifically includes: Update the position and covariance matrix of the non-overlapping regions in the background Gaussian model at the current time step using the background point cloud data at the current time step; The spherical harmonic coefficients and opacity in the background Gaussian model at the current time step are updated with the background image.

5. The autonomous driving decision-making method based on 3D scene reconstruction and multimodal large model according to claim 3, characterized in that, Integrating the dynamic object Gaussian map into the background Gaussian model to obtain the road condition Gaussian field at the current time step specifically includes: The opacity of the Gaussian map of the dynamic object is determined based on the distance between the Gaussian map of the dynamic object and the coordinates of the camera center. Using the adjusted opacity as the center of the Gaussian distribution of the dynamic object, calculate the transformation matrix of the Gaussian distribution of the dynamic object to the background Gaussian model. The Gaussian map of the dynamic object is stitched into the background Gaussian model in chronological order using a transformation matrix to obtain the Gaussian field of the road condition.

6. The autonomous driving decision-making method based on 3D scene reconstruction and multimodal large model according to claim 1, characterized in that, The step of rendering the Gaussian field of the road condition based on multi-view images to obtain the dynamic driving scene at the current time step specifically includes: The road condition Gaussian field is mapped onto a two-dimensional plane using a 3D Gaussian splash renderer to obtain the road condition plane image.

Images