A perception representation stitching method and system for end-to-end autonomous driving systems

Abstract

The application provides a perception representation stitching method and system for an end-to-end autonomous driving system, the method comprising: obtaining a paired observation dataset, each sample in the paired observation dataset containing target observation and source observation under the same environment state; encoding the target observation and the source observation by using a target perception encoder to obtain target latent representation and source latent representation; learning a stitching mapping from the source latent representation to the target latent representation; deploying the stitching mapping between the output end of the source perception encoder and a target downstream model, and freezing the parameters of the target downstream model; inputting the current representation obtained by encoding real-time observation by the source perception encoder into the stitching mapping, and inputting the transformed representation into the target downstream model to generate perception results or decision control instructions of autonomous driving. The application does not need to retrain the downstream strategy, and only by deploying a low-complexity stitching mapping, most of the downstream driving performance can be restored under most update scenarios.

Description

Technical Field

[0001] This application relates to the field of autonomous driving systems and deep learning model adaptation, specifically to a method and system for stitching perception representations for end-to-end autonomous driving systems. Background Technology

[0002] In existing end-to-end or perception-decision joint optimization autonomous driving systems, the upstream perception module (including different network skeletons, different sensor configurations, or different training strategies) is frequently updated to improve perception performance. The corresponding downstream decision model usually has high training costs and long validation cycles, so in engineering practice it is often not retrained with each perception update.

[0003] The comparison techniques and their limitations include: Full retraining: Retraining the updated perception output and downstream policies from scratch can restore performance, but it has high computational and data costs and a long verification cycle, making it unsuitable for frequent iteration scenarios.

[0004] Fine-tuning: Fine-tuning all parameters of the downstream model can lead to faster convergence, but it still requires significant computational / data resources and carries the risk of behavioral breakdown.

[0005] Adapter (Parameter-efficient tuning): Inserts a lightweight module into the policy network and fine-tunes it. The parameters are efficient, but it will change the internal representation of the policy. The adaptability is limited by the magnitude of the representation change.

[0006] Jointly trained or complex adaptation modules (variants such as DriveAdapter): require joint training or complex design, have high engineering complexity, and have limited adaptability when there are significant changes in cross-domain or sensor configuration.

[0007] A search revealed that CN202310811453.1 discloses an end-to-end training method based on a teacher-student architecture, which improves model accuracy by jointly optimizing sensing and control signals. However, this method relies on global parameter updates. When the upstream sensing architecture is replaced, sensors are changed, or cross-domain iterations occur, the downstream decision network still needs to be fully retrained or fine-tuned. This method suffers from drawbacks such as high computational cost, long verification cycle, and inability to quickly adapt when the downstream model is frozen. Summary of the Invention

[0008] In view of one of the deficiencies in the prior art, the purpose of this application is to provide a perception representation stitching method and system for an end-to-end autonomous driving system.

[0009] A first aspect of this application provides a perception representation stitching method for an end-to-end autonomous driving system, comprising: Obtain a paired observation dataset, wherein each sample in the paired observation dataset contains target observations and source observations under the same environmental conditions; The target observations are encoded using a target-aware encoder to obtain a potential target representation; the source observations are encoded using a source-aware encoder to obtain a potential source representation. Based on the target latent representation and the source latent representation, a stitching mapping from the source latent representation to the target latent representation is learned; The stitching map is deployed between the output of the source-aware encoder and the target downstream model, and the parameters of the target downstream model are frozen. The current representation obtained by encoding real-time observations by the source perception encoder is input into the stitching map, and the transformed representation is input into the target downstream model to generate perception results or decision control instructions for autonomous driving.

[0010] Optionally, when the stitching mapping is a linear stitching mapping, it is suitable for vectorized latent representations, and its transformation function is expressed as follows: ,in The character to be transformed, It is a linear transformation matrix. This is the bias vector.

[0011] Optionally, the learning of the stitching mapping from the source latent representation to the target latent representation includes: The target latent representation and the source latent representation are stacked in the sample row direction to construct the target representation matrix Z1 and the source representation matrix Z2; Based on the target representation matrix Z1 and the source representation matrix Z2, the linear transformation matrix is ​​solved using the least squares method or ridge regression with regularization. and the bias vector The closed-form solution yields the linear stitching mapping, specifically: ; in, Z1 represents the mean of all samples. Z2 represents the mean of all samples.

[0012] Optionally, when the stitching mapping is a convolutional stitching mapping, it is suitable for high-dimensional convolutional feature map representation, and its transformation function is expressed as follows: ,in The feature map to be transformed. The parameters are 1×1 convolution kernel parameters. This indicates batch normalization operation.

[0013] Optionally, the learning of the stitching mapping from the source latent representation to the target latent representation includes: Freeze the parameters of the source-aware encoder and the target downstream model; Construct the total loss function ,in The task supervision loss is based on the output of the target downstream model. The feature alignment loss is used to measure the difference between the transformed representation and the target potential representation, where λ≥0 is the weighting coefficient. Based on the paired observation dataset and the total loss function, the parameters of the 1×1 convolution kernel and / or the parameters of the batch normalization operation are optimized by the gradient descent algorithm to obtain the convolution stitching map.

[0014] Optionally, the paired observation dataset is constructed in at least one of the following ways: Simultaneous data collection using different sensor configurations or different perception models under the same driving scenario; In a simulation environment, observation data with different perspectives, modes, or external parameters are generated based on the same physical state. Timestamp alignment and sampling are performed on historical road test data that spans different regions, weather conditions, or calibration conditions.

[0015] Optionally, the method can be applied to at least one of the following upstream sensing system update scenarios: Replacement of the perceptual encoder network architecture; The number of sensors increases or decreases, or their installation orientation changes; Training dataset distribution transfer; Multimodal input configuration change.

[0016] A second aspect of this application provides a perception representation stitching system for an end-to-end autonomous driving system, comprising: The data acquisition module acquires a paired observation dataset, wherein each sample in the paired observation dataset contains a target observation and a source observation under the same environmental condition; The representation encoding module encodes the target observations using a target-aware encoder to obtain a potential target representation; and encodes the source observations using a source-aware encoder to obtain a potential source representation. The mapping learning module learns a stitching mapping from the source latent representation to the target latent representation based on the target latent representation and the source latent representation; The deployment module deploys the stitching mapping between the output of the source-aware encoder and the target downstream model, and freezes the parameters of the target downstream model. The inference module inputs the current representation obtained by the source perception encoder from real-time observations into the stitching map, and inputs the transformed representation into the target downstream model to generate perception results or decision control commands for autonomous driving.

[0017] A third aspect of this application provides a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to execute any of the perception representation stitching methods for end-to-end autonomous driving systems described in the previous application.

[0018] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform any of the perception representation stitching methods for end-to-end autonomous driving systems described in the present invention.

[0019] The perception representation stitching method for end-to-end autonomous driving systems provided in this application can restore most of the downstream driving performance in most update scenarios without retraining the downstream policy, simply by deploying a low-complexity stitching mapping.

[0020] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0021] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a perception representation stitching method for an end-to-end autonomous driving system according to an exemplary embodiment; Figure 2 This is a flowchart illustrating a perception representation stitching method for an end-to-end autonomous driving system according to an exemplary embodiment. Figure 3 This is a structural diagram of a perception representation stitching system for an end-to-end autonomous driving system, according to an exemplary embodiment. Detailed Implementation

[0022] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0023] Existing technologies generally cannot quickly restore downstream behavior compatibility with low computational and data costs while "freezing the downstream decision-making model without changing its internal parameters". Based on the above problems, this application provides a perception representation stitching method for an end-to-end autonomous driving system to solve the aforementioned issues.

[0024] Reference Figure 1 and Figure 2 As shown, in one embodiment of this application, a perception representation stitching method for an end-to-end autonomous driving system is provided, comprising the following steps: S100, Obtain the paired observation dataset. Each sample in the paired observation dataset contains target observations and source observations under the same environmental conditions. S200: The target observation is encoded using a target-aware encoder to obtain the target latent representation; the source observation is encoded using a source-aware encoder to obtain the source latent representation. S300, based on the target latent representation and the source latent representation, learns the stitching mapping from the source latent representation to the target latent representation; S400 deploys the stitching map between the output of the source-aware encoder and the target downstream model, and freezes the parameters of the target downstream model; The S500 inputs the current representation obtained by encoding real-time observations from the source perception encoder into a stitching mapping, and inputs the transformed representation into the target downstream model to generate perception results or decision control commands for autonomous driving.

[0025] The above embodiments introduce low-complexity stitching mapping at the representation interface, enabling rapid alignment of new perceptual representations with existing decoders / policies in the feature space, thereby avoiding retraining or large-scale fine-tuning. After the upstream perceptual model is replaced or updated, the performance of downstream tasks (such as path completion rate, driving score, and perception mIoU / mAP) can be quickly restored or maintained with frozen downstream model parameters, requiring only a very small amount of pairing data and extremely low computational overhead, effectively solving the problems of high adaptation costs and long iteration cycles of traditional solutions.

[0026] In one specific embodiment of this application, S100, a paired observation dataset is obtained, wherein each sample in the paired observation dataset contains target observations and source observations under the same environmental state. The specific implementation process is as follows: Based on the same environmental state Paired target observations are simultaneously acquired or simulated and generated through the target observation generation path and the source observation generation path, respectively. Source observation ; Among them, environmental status The system represents the complete physical or simulated scene in which it exists (including vehicle pose, distribution of traffic participants, road topology, and high-precision map features); target observation. Source observation To map raw observation data (such as multi-view images, LiDAR point clouds, or preprocessed tensors fused from both) acquired from different sensor hardware configurations, different sensing network architectures, or different training data distributions under the same spatiotemporal reference, using a state mapping function... ,in In order to map the state to the observation result, a strict one-to-one correspondence is required in the physical scene.

[0027] Based on the strictly spatiotemporally aligned paired samples of the above embodiments, the stitching mapping can accurately capture the distribution mapping relationship between different representation spaces. Therefore, even if there is a network architecture replacement or only a difference in initialization parameters between the source sensing encoder and the target encoder, the downstream decision model after stitching can still maintain a stable feature input distribution, and its downstream task performance can be restored to a level close to the original system baseline, effectively avoiding significant degradation in decision performance caused by sensing module iteration.

[0028] Furthermore, in one specific implementation, the paired observation dataset can be constructed in at least one of the following ways: (1) Use different sensor configurations or different perception models to collect data synchronously in the same driving scenario.

[0029] For example, two sets of sensing hardware or algorithm links, one old and one new, are installed in parallel on the same test platform and travel along a preset route. The timing of data acquisition is ensured by aligning hardware synchronization signals or timestamps, and the observation stream is captured at the same frame rate to directly form a pair of target observations and source observations in the same traffic scenario.

[0030] (2) Generate observation data with different perspectives, modes or external parameters based on the same physical state in the simulation environment.

[0031] For example, the same scene state and traffic flow distribution can be locked in the simulation platform, and different sensor blueprints or view parameters can be called to render independently, and strictly corresponding target observations and source observations can be generated synchronously, thereby obtaining paired data of the same source across viewpoints / modalities.

[0032] (3) Perform timestamp alignment and sampling on historical road test data that cross regions, weather conditions, or calibration conditions.

[0033] For example, scene similarity matching and spatiotemporal alignment techniques can be used to extract structurally consistent observation segments from historical driving data under different geographical or environmental conditions; or the same batch of original observation data can be reprocessed offline using different calibration parameters or algorithm configurations to generate cross-domain or cross-version paired observations.

[0034] The paired observation dataset constructed using the above method possesses strict spatiotemporal consistency and high signal-to-noise ratio characteristics.

[0035] In one specific embodiment of this application, in step S200, the target observation is encoded using a target-aware encoder to obtain a potential target representation; the source observation is encoded using a source-aware encoder to obtain a potential source representation. This can be achieved through the following steps: Target observation Input to target perception encoder 1. Obtain the potential characterization of the target z1; then, use the source observations... Input to source sensing encoder 2. Obtain the source potential characterization z2; Among them, the target-aware encoder and the source-aware encoder are both pre-trained deep network models used to map the observation space Ω to the latent representation space Z; the latent representation can be a one-dimensional vector or a high-dimensional spatial feature map, used to carry high-level semantic information in the corresponding observation environment.

[0036] Based on the representation extraction results of the above embodiments, if the perception model remains unchanged and only the number and pose of sensors change, the decision model with the new perception network stitched together can maintain more than 98% of the original performance.

[0037] In one specific embodiment of this application, the method of this application is applicable to handling perceptual representation offset scenarios of varying degrees, including but not limited to initialization differences, network architecture replacement, sensor / modal configuration changes, and cross-data domain distribution differences.

[0038] For example, initialization differences: When the target encoder and the source encoder use the same network structure, the output representation space may have differences in numerical distribution, such as translation, scaling or channel sorting, only due to different random seeds or pre-trained weight initialization. For example, architecture replacement: When the perceptual backbone network is replaced with a model with a different topology or feature extraction mechanism (such as swapping CNN and Transformer, or replacing with different depth / width variants), the underlying representation will undergo structural changes in the channel dimension and high-level semantic distribution due to the different receptive field design and semantic abstraction path. For example, modal / sensor changes: When the vehicle hardware configuration is adjusted (such as increasing or decreasing the number of cameras, shifting the pitch / altitude of the installed extrinsic parameters, or switching between image and LiDAR modes), the spatial coverage, resolution, and physical perception dimension of the input data change, resulting in geometric or modal misalignment of the representation space distribution; For example, cross-domain differences: when a model needs to be migrated from the source data domain to the target deployment domain, changes in external environmental factors such as lighting conditions, weather interference, geographical topology, and the distribution of traffic participants can lead to inconsistencies between the prior data distribution in the source representation and the target environment.

[0039] The above embodiments demonstrate that the characterization stitching technique provided in this application can flexibly adapt to various offset scenarios, from lightweight parameter drift to heavy-duty architecture / hardware iteration, and has broad engineering applicability and scenario generalization capability.

[0040] Different representational offsets differ fundamentally in their inherent mathematical properties, spatial topology, and the complexity of their distribution variations. A single stitching strategy cannot simultaneously guarantee adaptation accuracy, convergence stability, and compatibility with downstream strategies in all scenarios. Therefore, different robust stitching strategies are required. Stitching Mapping The optimization objective can be formally defined as:

[0041] in, For paired observation datasets, (,) represents the behavioral distance metric function in the downstream decision / action space. For the target downstream model with frozen parameters, and These are the target sensing encoder and the source sensing encoder, respectively. The preset performance tolerance threshold is defined by this formula. This formula indicates that the learning of the stitching mapping aims to minimize the expected behavioral difference between the decision output generated in the downstream model after encoding and transformation of the source observation and the original target observation decision output, and to constrain it within the tolerance range, thereby mathematically ensuring seamless compatibility and behavioral consistency of the downstream strategy. Based on the above unified optimization objective, this application provides two types of low-complexity mappings for solving different types of latent representations: linear stitching and convolutional stitching.

[0042] In one specific embodiment of this application, when the latent representation is in vectorized form (i.e., the latent representation is in vectorized form), it means that the feature data output by the perceptual encoder has been removed from the two-dimensional spatial structure (height H, width W) and compressed or flattened into a mathematical form of a one-dimensional numerical array, usually represented as z∈R n When this is done, linear stitching mapping is used for adaptation; Specifically, the transformation function of the linear stitching map is defined as follows: ,in As a latent representation of the input source, It is a linear transformation matrix. This is the bias vector.

[0043] Furthermore, based on the paired observation dataset, m The target latent representations and source latent representations of each sample are stacked row-wise to construct the target representation matrix. With source representation matrix ; Solve the linear transformation matrix using the least squares method or the ridge regression algorithm with regularization. With bias vector Closed-form solution: ; The linear stitching scheme described above directly obtains the optimal mapping parameters through a closed-form analytical solution, without iterative optimization. The solution process is deterministic and computationally efficient; its parameter count is only a few. O ( n 2) It has low requirements for the size of paired samples and can effectively cope with the near-linear shifts in the representation space distribution, such as translation, scaling and channel rearrangement, so as to achieve near real-time deployment readiness while ensuring the accuracy of adaptation.

[0044] In some specific embodiments of this application, when the potential representation is in the form of a high-dimensional convolutional feature map (referring to multidimensional tensor data that retains the two-dimensional spatial topology and is directly output by the perceptual encoder in the intermediate layer or at the end), convolutional stitching mapping is used for adaptation.

[0045] Specifically, the transformation function of the convolution stitching map is defined as follows: ,in: For the input source feature map, A 1×1 convolution kernel ( BN represents batch normalization.

[0046] During training, the parameters of the source-aware encoder and the target downstream model are frozen, and only the 1×1 convolution kernel parameters and batch normalization parameters are updated. Construct the total loss function

[0047] in λ ≥0 indicates an alignment weight coefficient. Task monitoring loss Feature alignment loss They are defined as follows:

[0048]

[0049] in, Let i be the labeled true value of the i-th sample. Frobenius norm This is the loss function for downstream tasks (such as cross-entropy or mean squared error). It is a strategy model. These are the network parameters of the policy model. It's a convolutional stitching mapping. p is the perceptual model. These are the network parameters of the perceptual model. o is the perceptual data, y is the label, and m is the total number of samples.

[0050] The convolutional stitching scheme in the above embodiments performs feature remapping in the channel dimension through 1×1 convolution, which completely preserves the spatial topology of the feature map and avoids the destruction of spatial information caused by fully connected mapping. Its parameter count is only O(Cin×Cout), which is significantly lower than that of global affine transformation. It can effectively cope with complex nonlinear offsets such as sensor extrinsic parameter shift, channel response misalignment and cross-domain distribution changes. Combined with the dual optimization mechanism of task supervision and feature alignment, it can achieve fast convergence in very few training rounds, taking into account both adaptation accuracy and engineering deployment efficiency.

[0051] In some specific embodiments of this application, in S400, the stitching map is deployed between the output of the source-aware encoder and the target downstream model, and the parameters of the target downstream model are frozen. This can be implemented in the following way: The stitching map is deployed between the feature output of the source-aware encoder and the input of the target downstream model (such as a decoder or policy network). Under this deployment architecture, the original weight parameters of the target downstream model remain frozen and do not participate in gradient updates.

[0052] After deployment, the entire model with stitch mapping added (which includes the source-aware encoder, stitch mapping layer, and target downstream model) needs to be trained. The training process is as follows: First, construct a paired sample dataset. Parameter optimization for stitching mapping. Paired samples can be obtained by synchronously running two sets of different sensor configurations or encoders with different architectures in the same physical scene, or by synchronously recording in a high-fidelity simulation environment combined with a real scene. The sample size of the dataset is adaptively configured according to the complexity of the downstream task and the network complexity of the stitching mapping. For example, for linear stitching mapping, hundreds to thousands of paired samples are usually sufficient to meet the optimization convergence requirements.

[0053] Next, differential parameter solving is performed based on the specific structural type of the stitching map.

[0054] For linear stitching mapping, the linear transformation weight matrix is ​​solved directly by constructing a distribution alignment objective function between the source features and the target features, and using an analytical closed-form solution or a fast batch regression algorithm. For the convolutional stitching mapping, mini-batch gradient descent optimization (using SGD or Adam optimizer) is performed based on the paired sample dataset. In one specific implementation, the training hyperparameters are configured such that the learning rate is set to 1×10⁻⁶. 3. Set the batch size to 16, configure the number of training rounds from 1 to 10 depending on the data size, and introduce a batch normalization module between network layers to accelerate gradient convergence and stabilize feature distribution.

[0055] Finally, after the stitching mapping is deployed and optimized, downstream decision-making task metrics and front-end perception task metrics are used as the core evaluation dimensions. Downstream decision-making metrics include path completion rate (RC) and driving safety score (DS); perception metrics include mean intersection-over-union ratio (mIoU) and mean average accuracy (mAP). By comparing the stitching mapping strategy of this application with various baseline schemes (including direct feature connection, linear stitching, convolutional stitching, adapter fine-tuning, full parameter fine-tuning, and model retraining from scratch), the effectiveness of each strategy in restoring the performance of downstream models and its corresponding computational resource overhead and training time cost are comprehensively evaluated.

[0056] Based on the same technical concept, other embodiments of this application provide a perception representation stitching system 100 for an end-to-end autonomous driving system, such as... Figure 3 As shown, it includes: Data acquisition module 110 acquires paired observation datasets, where each sample contains target observations and source observations under the same environmental conditions; The representation encoding module 120 uses a target-aware encoder to encode target observations to obtain a potential target representation; and uses a source-aware encoder to encode source observations to obtain a potential source representation. The mapping learning module 130 learns a stitching mapping from the source latent representation to the target latent representation based on the target latent representation and the source latent representation; Deployment module 140 deploys the stitching map between the output of the source-aware encoder and the target downstream model, and freezes the parameters of the target downstream model; The inference module 150 inputs the current representation obtained by encoding real-time observations from the source perception encoder into the stitching mapping, and inputs the transformed representation into the target downstream model to generate perception results or decision control commands for autonomous driving.

[0057] The specific implementation techniques of each module / unit in the above examples of this application can be referred to the steps of the perception representation stitching method for end-to-end autonomous driving system in the above embodiments, and will not be repeated here.

[0058] The above embodiments offer high efficiency in restoring downstream behavior compatibility, eliminating the need to retrain downstream strategies. A low-complexity stitching module can restore most downstream driving performance in most update scenarios. They feature extremely low computational and data costs; linear stitching can be quickly solved using a closed-form solution; convolutional stitching parameters are much smaller than full fine-tuning, resulting in significantly shorter training time than retraining / fine-tuning, and low GPU and memory usage. They are highly deployable, with simple stitching layer insertion points, clear implementation steps, and rapid execution at the edge or in the cloud, facilitating frequent iterations of the perception model in production environments without affecting the long-term validation process of the decision model. They support both support vector and feature map representations and are suitable for various update scenarios such as architecture replacement, modality changes, and cross-domain migration. They are interpretable and verifiable; linear stitching has a closed-form solution, facilitating the analysis of transformation properties; task supervision and feature alignment loss can quantify stitching quality.

[0059] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0060] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0061] Example 1: Linear stitching for adapting AE encoders to VAE encoders In this embodiment, the target model employs a VAE-based encoder. The source model uses an AE encoder and is connected to a downstream decoder or policy network. The source model and the target model have different encoder structures, but the corresponding perceptual model supervision signals are of the same type.

[0062] Specifically, sampling under the same or corresponding driving scenarios. For paired latent vectors and ,in By target model encoder Target observation Encoded to obtain, Source model encoder Source observation Obtained through encoding. For and After performing mean centering, the linear mapping matrix is ​​solved using ridge regression. The regularization parameter is selected using cross-validation. Furthermore, the bias is calculated based on the mean term. Thus, the linear stitching module is obtained:

[0063] During the deployment phase, the aforementioned linear stitching module will be used. The linear stitching module directly inserts itself between the source model encoder output and the target model downstream input, freezing the parameters of the target model's downstream decoder or policy network, and completing the latent space adaptation solely through this linear stitching module. Subsequently, the driving performance of the target model is tested in closed-loop or offline driving evaluation tasks to verify the effectiveness of this linear stitching method in scenarios involving the migration from an AE encoder to a VAE encoder.

[0064] Example 2: Convolution Stitching under Scenarios of Varying Sensor Number and Pose In this embodiment, the source model is configured with six cameras and one LiDAR, while the target model is configured with four cameras and one LiDAR. Because the source and target models differ in the number of sensors, camera field of view coverage, and sensor mounting poses, the intermediate features generated by them are not entirely consistent in channel response and spatial distribution. Therefore, a convolutional stitching module is used to adapt the features of the source model.

[0065] Specifically, a corresponding driving scenario is constructed in the CARLA simulation environment, and observations are collected under the source model sensor configuration. And observations under the target model sensor configuration Source observation Input source model encoder The intermediate features of the source model are obtained:

[0066] At the same time, target observation can be performed. Input target model encoder Record the intermediate features of the target model:

[0067] As a reference for feature alignment.

[0068] In the source model, intermediate features Downstream network of the target model Insert a 1×1 convolution stitcher between them This convolutional stitcher is used to remap the channel representations of the source model's output features without altering the feature map spatial resolution. In one specific implementation, The input channel count is 32, the output channel count is 32, and the kernel size is 1×1. The feature representation after convolution stitching is as follows:

[0069] Then, Input target model downstream network This is to obtain outputs related to BEV segmentation, object detection, or driving decisions. During training, the source model encoder is frozen. and downstream network of the target model The parameters are used to train only the convolutional stitcher. The training objective uses task-supervised loss. For example, BEV segmentation labels can be used as supervision signals for optimization. Feature alignment loss can be introduced. Used for measurement Reference features of the target model The differences between them further improve the consistency of features after suturing. The total loss can be expressed as:

[0070] in These are the weighting coefficients.

[0071] In the training configuration, the learning rate (lr) is set to 1e-3, the batch size to 16, and the number of training epochs to 1, to achieve rapid adaptation; the number of training epochs can also be increased according to the target accuracy requirements. After training is complete, the source model encoder is retained. Convolutional stitcher and downstream network of the target model The connections between the sensors form a new inference link. During the deployment phase, this inference link is evaluated, with evaluation metrics including but not limited to mIoU, RC, and DS. When the above metrics meet preset thresholds, the convolutionally stitched model link can replace the original online inference link for autonomous driving perception or decision-making tasks after changes in the number and pose of sensors.

[0072] Example 3: Convolution Stitching in Scenarios with Changing Sensor Pose and Data Domain In this embodiment, the source model is trained on the NuScenes dataset or other real-world road datasets, while the target model is designed for a different target data domain or deployment environment. The source and target models may have the same or similar number of sensors, but they differ in sensor extrinsic parameters, installation height, pitch angle, field of view coverage, road scene distribution, lighting conditions, weather conditions, and data acquisition equipment calibration methods. These differences can lead to inconsistencies between the intermediate features output by the source model and the feature distribution expected by the downstream network of the target model. Therefore, a convolutional stitching module can be used for cross-pose and cross-data domain adaptation.

[0073] The specific suturing process is exactly the same as in Example 2.

[0074] Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-described perception representation stitching method for an end-to-end autonomous driving system.

[0075] Based on the same technical concept, in other embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to perform the above-described perception representation stitching method for an end-to-end autonomous driving system.

[0076] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0077] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0078] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0079] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0080] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A method for stitching perception representations together in an end-to-end autonomous driving system, characterized in that, include: Obtain a paired observation dataset, wherein each sample in the paired observation dataset contains target observations and source observations under the same environmental conditions; The target observations are encoded using a target-aware encoder to obtain a potential target representation; the source observations are encoded using a source-aware encoder to obtain a potential source representation. Based on the target latent representation and the source latent representation, a stitching mapping from the source latent representation to the target latent representation is learned; The stitching map is deployed between the output of the source-aware encoder and the target downstream model, and the parameters of the target downstream model are frozen. The current representation obtained by encoding real-time observations by the source perception encoder is input into the stitching map, and the transformed representation is input into the target downstream model to generate perception results or decision control instructions for autonomous driving.

2. The perception representation stitching method for an end-to-end autonomous driving system according to claim 1, characterized in that, When the stitching mapping is a linear stitching mapping, it is suitable for vectorized latent representations, and its transformation function is expressed as follows: ,in The character to be transformed, It is a linear transformation matrix. This is the bias vector.

3. The method according to claim 2, characterized in that, The learning of the stitching mapping from the source latent representation to the target latent representation includes: The target latent representation and the source latent representation are stacked in the sample row direction to construct the target representation matrix Z1 and the source representation matrix Z2; Based on the target representation matrix Z1 and the source representation matrix Z2, the linear transformation matrix is ​​solved using the least squares method or ridge regression with regularization. and the bias vector The closed-form solution yields the linear stitching mapping, specifically: ； in, Z1 represents the mean of all samples. Z2 represents the mean of all samples.

4. The perception representation stitching method for an end-to-end autonomous driving system according to claim 1, characterized in that, When the stitching mapping is a convolutional stitching mapping, it is suitable for high-dimensional convolutional feature map representation, and its transformation function is expressed as follows: ,in The feature map to be transformed. The parameters are 1×1 convolution kernel parameters. This indicates batch normalization operation.

5. The method according to claim 4, characterized in that, The learning of the stitching mapping from the source latent representation to the target latent representation includes: Freeze the parameters of the source-aware encoder and the target downstream model; Construct the total loss function ,in The task supervision loss is based on the output of the target downstream model. The feature alignment loss is used to measure the difference between the transformed representation and the target potential representation, where λ≥0 is the weighting coefficient. Based on the paired observation dataset and the total loss function, the parameters of the 1×1 convolution kernel and the parameters of the batch normalization operation are optimized by the gradient descent algorithm to obtain the convolution stitching mapping.

6. The perception representation stitching method for an end-to-end autonomous driving system according to claim 1, characterized in that, The paired observation dataset is constructed in at least one of the following ways: Simultaneous data collection using different sensor configurations or different perception models under the same driving scenario; In a simulation environment, observation data with different perspectives, modes, or external parameters are generated based on the same physical state. Timestamp alignment and sampling are performed on historical road test data that spans different regions, weather conditions, or calibration conditions.

7. The perception representation stitching method for an end-to-end autonomous driving system according to claim 1, characterized in that, The method is applied to upstream sensing system update scenarios with varying degrees of representational offset, which includes initialization differences, architecture replacement, modality or sensor changes, and cross-domain differences.

8. A perception representation stitching system for an end-to-end autonomous driving system, characterized in that, include: The data acquisition module acquires a paired observation dataset, wherein each sample in the paired observation dataset contains a target observation and a source observation under the same environmental condition; The representation encoding module encodes the target observations using a target-aware encoder to obtain a potential target representation; and encodes the source observations using a source-aware encoder to obtain a potential source representation. The mapping learning module learns a stitching mapping from the source latent representation to the target latent representation based on the target latent representation and the source latent representation; The deployment module deploys the stitching mapping between the output of the source-aware encoder and the target downstream model, and freezes the parameters of the target downstream model. The inference module inputs the current representation obtained by the source perception encoder from real-time observations into the stitching map, and inputs the transformed representation into the target downstream model to generate perception results or decision control commands for autonomous driving.

9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the perception representation stitching method for an end-to-end autonomous driving system as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to perform the perception representation stitching method for an end-to-end autonomous driving system as described in any one of claims 1-7.

Images