Automatic driving method with cross-domain data fusion and double-space perception enhancement

Abstract

The invention discloses an automatic driving method with cross-domain data fusion and double-space perception enhancement. The automatic driving method comprises the following steps: extracting multi-modal features; constructing a cross-domain data fusion strategy; establishing a double-space perception enhancement module; and performing end-to-end trajectory planning and reasoning. Through dynamic weight distribution based on data set difference and dynamic fine adjustment based on loss feedback, the training weights of conventional and long-tail scenes are balanced, the basic capability of the conventional scene and the generalization capability of the edge scene are cooperatively improved, and the robustness of the system is remarkably improved; according to the invention, a double-space perception enhancement mechanism is provided, and a bird's-eye view global structure and look-around local details are decoupled and fused; according to the method, the characteristics of a long-distance small target are accurately captured by utilizing a self-adaptive cutting technology guided by a multi-modal pre-training model based on comparative learning, so that the problem of information loss caused by insufficient spatial resolution of a single aerial view is solved, and the sensing precision of key targets such as traffic signs is improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an autonomous driving method with cross-domain data fusion and dual-space perception enhancement. Background Technology

[0002] End-to-end autonomous driving systems aim to achieve a direct mapping from sensor input to driving decisions. However, existing technologies face significant challenges when dealing with complex traffic environments. First, existing models lack robustness in long-tail scenarios (such as extreme weather and irregularly shaped obstacles). This is mainly because training data is typically dominated by conventional scenarios, making it difficult for models to establish stable feature representations on scarce long-tail samples, and they are prone to catastrophic forgetting. Second, large models have limitations in spatial perception and multi-spatial information coupling. While bird's-eye views provide global spatial structure, their resolution often limits the loss of detailed information about distant small targets (such as traffic cones and road signs). While surround-view images are rich in local texture, they lack global topology. Existing methods struggle to effectively balance and fuse the features of these two spaces, resulting in low small target recognition rates. In conclusion, dynamically balancing the learning weights of different scenarios through cross-domain data fusion strategies and establishing a dual-spatial perception enhancement mechanism to deeply fuse global structure and local details, thereby outputting driving plans that are both robust and accurate, has become an important research topic. Summary of the Invention

[0003] To address the aforementioned problems in existing technologies, this invention proposes an autonomous driving method with cross-domain data fusion and dual-space perception enhancement. The method dynamically optimizes the weights of training samples through a cross-domain data fusion strategy and utilizes a dual-space perception enhancement mechanism to achieve deep coupling between global and local features, thereby outputting a highly robust driving trajectory.

[0004] To achieve the above objectives, the specific technical solution of the present invention is as follows: an autonomous driving method with cross-domain data fusion and dual-space perception enhancement, comprising the following steps: A. Extracting multimodal features A1. Extracting visual features First, given a set to be generated In each camera view, since each front view image is associated with a different camera pose, therefore, The multi-frame, multi-view images from a single camera are represented as follows:

[0005] In the formula, , and They are the first The intrinsic parameters, extrinsic rotation matrix, and translation vector of each camera; Number of cameras; The number of historical timestamps during the vehicle's operation; , 3 and represent the length, width, and number of channels of the acquired image, respectively. Each RGB image is fed into the feature extraction network to generate multi-view features, represented as:

[0006] In the formula, This represents the number of channels.

[0007] A2. Extracting Text Features Language information is divided into several task-oriented subsets, with environmental perception and decision-making / planning subsets as typical examples, and a language information set is defined. The environmental perception subsequence and the decision-making and planning subsequence are respectively denoted as... and The hierarchical representation of language information is as follows:

[0008] in, The number of language-related tasks; The number of historical timestamps is the same as the timestamps of the image sequence. To transform discrete language symbols into continuous semantic features for model processing, word embedding is first performed on each language text information, mapping it to a low-dimensional dense vector; subsequently, instruction temporal encoding vectors are introduced. Ultimately, a unified dimension of language features is obtained. The formula is as follows:

[0009] In the formula, For the first Language-like modalities in the first The original linguistic information of the moment; and These represent the instruction sequence length and the text feature dimension, respectively. This is a word embedding function.

[0010] B. Constructing a cross-domain data fusion strategy The cross-domain data fusion strategy divides the training process into two dynamic optimization phases, spanning the entire iterative training cycle of the end-to-end autonomous driving model. The first phase is an active weight allocation phase based on dataset differences. This phase calculates static cross-domain redundancy between datasets based on the inherent differences in feature space across different datasets, thereby establishing the initial sampling priority for various scenarios. Based on this, training batches are constructed through weighted sampling and input into the end-to-end autonomous driving model for training and loss calculation. The second phase is a delay adjustment phase based on loss feedback. This phase introduces dynamic loss changes during training as feedback signals to perform secondary fine-tuning of weights for scenarios where learning stagnates. Through the synergistic effect of these two phases, balanced training for both conventional and long-tail scenarios is achieved. The specific steps are as follows: B1. First Stage: Assigning Dynamic Weights Based on Dataset Differences First, define the training data set. To accurately measure the characteristic attributes of various scenarios, modal feature representation is adopted, that is, visual features are used. and text semantic features This indicates that visual features and text semantic features The weighted fusion yields:

[0011] In the formula, For scene feature vectors; For dataset The number of samples; For feature fusion weights. For dataset Visual features; For dataset Textual features.

[0012] To eliminate the size differences in different modal characteristics, Normalization is performed, and the normalized scene feature vector is calculated using the following formula. :

[0013] In the formula, It is an L2 norm.

[0014] Next, the cross-domain redundancy between different datasets is quantified for any two datasets. and The difference between the two is expressed as Further calculations on a single dataset Average difference with all other datasets The formula is as follows:

[0015] In the formula, The total number of data sets; For dataset and dataset Differences; This represents the average difference between datasets.

[0016] Then, based on the average difference Update sampling weights Every preset training step length The sampling weights are adjusted once based on the differences in the dataset. Based on this, the sampling weights are then... In the dataset at each training step size The weight iterative update function is expressed as:

[0017] In the formula, For the first Training dataset in rounds The sampling weights, which are only applied during training epochs. To update step size Update when the value is an integer multiple of the total value. This is the parameter for adjusting the weight. This represents the scene preference coefficient. To suppress extreme value fluctuations, improve training stability, and ensure that the weights conform to the probability distribution, the updated weights are... After smoothing and normalization, it is represented as:

[0018]

[0019] In the formula, For the smoothed first A dataset in Weights for training step size; For smoothing terms; For the smoothed first A dataset in Weights for training step size; For the normalized first A dataset in Weights for training step size.

[0020] Based on the above calculations, the normalized sampling weight vector The end-to-end autonomous driving model training iteration is performed, and the specific steps are as follows: B11, Data Sampling: In the... During step training, based on the sampling weight vector From the data set Randomly selected from the middle Training batch of samples.

[0021] B12. Forward Propagation: The sampled multimodal data is input into the subsequent step CD to obtain a joint output set containing environmental perception interpretation and trajectory planning. .

[0022] B13. Loss Function Calculation: Define the total training loss. It includes environmental perception explanation loss. Multi-view feature alignment loss and trajectory planning loss Regarding the current time step Calculate the average loss value .

[0023] B14, Parameter Update: Utilizing The parameters are updated via backpropagation, and the specific loss for each dataset is recorded. As a feedback signal, it is passed to the second stage for dynamic fine-tuning of the weights.

[0024] B2. Second Stage: Dynamic Fine-tuning Based on Loss Feedback To avoid false triggering caused by fluctuations in single-round losses, a hysteresis adjustment mechanism based on loss feedback is introduced. Let the... Step in dataset The training loss is Then the first Wheel relative to the first The percentage decrease in wheel losses is defined as:

[0025] In the formula, The constraints ensure that only the decrease in losses is included in the statistics; This is the loss from the previous round. If it continues... Each training step size satisfies the loss reduction ratio. Less than the preset threshold Then determine When a learning stagnation state is reached, a loss feedback adjustment factor is introduced. , is represented as:

[0026] In the formula, To adjust the sensitivity coefficient; The global average loss is used to reflect the first... The overall difficulty level of the round training; This is the overfitting risk coefficient. If stagnant learning occurs on some datasets, then... If it does not stop, then Since the application of the adjustment factor causes the sum of the weight vectors to deviate from 1, it is mapped to the probability distribution space, ultimately yielding the first... Round sampling weights ,Right now .

[0027] C. Establish a dual-space perception enhancement module C1, 3D visual environment perception By utilizing depth uncertainty estimation based on Gaussian sputtering, multi-view image features are projected onto 3D space and rasterized to generate global features for a bird's-eye view, as shown in the following formula:

[0028] In the formula, For the first The image domain of a camera; For the first Splash characteristics of individual cameras; For the first The transparency parameters of each camera; and The first Pixels per camera The mean and covariance on the bird's-eye view plane. To achieve dynamic object feature extraction and static map feature extraction from the global features of the bird's-eye view, three learnable semantic query vectors are introduced. , and These query vectors serve as semantic anchors for vehicles, pedestrians, and static maps, respectively. They interact with the bird's-eye view features through a cross-attention mechanism, adaptively focusing on the corresponding semantic regions. The final output consists of three types of semantic features. And all three belong to space.

[0029] C2. Extract fine-grained features from the surrounding view local area. Calculate the surround view image using a multimodal pre-trained model based on contrastive learning. With language and text information Cosine similarity is used to generate an attention heatmap. Identify local maxima in the attention heatmap and calculate the adaptive cropping window size. The formula is as follows:

[0030] in, To expand the coefficient, avoid cropping the edge details of small targets; For the first The first camera The pixel area of ​​a cropped image; Minimum window size to ensure feature richness. Centered on the centroid, with dimensions as cropped image The image encoder, re-input into a contrastive learning-based multimodal pre-trained model, extracts fine-grained features. And semantic information in sub-images.

[0031] C3. Perform deep fusion of dual spatial features To address the issue of missing fine-grained information in the feature space of bird's-eye view images, this paper achieves deep fusion of panoramic cropping features and global features of the bird's-eye view image, utilizing multi-view feature alignment loss. The consistency between the constraint clipping feature and the global features of the bird's-eye view is determined by the following formula:

[0032] In the formula, This represents the total number of cut blocks; These are the projection features of the bird's-eye view features at the corresponding locations; This represents the mean square error.

[0033] C4. Establish a cross-spatial attention module Based on the completion of multi-view feature semantic alignment, the global features of the bird's-eye view are used as the query. The aligned cropping features generate keys respectively. AND value Design a cross-spatial attention module and build and The complementary fusion mechanism injects local cropped features into the global features of the bird's-eye view, generating enhanced bird's-eye view features. :

[0034] D. Perform end-to-end trajectory planning and reasoning Enhanced bird's-eye view features Global features Local clipping features and text features The input to the Visual Language Model (VLM) is as follows:

[0035] In the formula, The output set is a joint set that includes environmental perception interpretation and future trajectory planning. For learnable projectors; It is a text segmenter.

[0036] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention designs a cross-domain data fusion strategy. By dynamically allocating weights based on dataset differences and dynamically fine-tuning based on loss feedback, it balances the training weights of conventional and long-tail scenarios, achieves synergistic improvement of basic capabilities in conventional scenarios and generalization capabilities in edge scenarios, and significantly improves the robustness of the system.

[0037] 2. This invention proposes a dual-spatial perception enhancement mechanism that decouples and fuses the global structure of the bird's-eye view with the local details of the surrounding view. Utilizing an adaptive cropping technique guided by a multimodal pre-trained model based on contrastive learning, it accurately captures the features of small targets at a distance, solving the problem of information loss caused by insufficient spatial resolution in a single bird's-eye view and improving the perception accuracy of key targets such as traffic signs. Attached Figure Description

[0038] This invention has a total of appendices Figure 4 Zhang, of which: Figure 1 This is a flowchart illustrating the present invention.

[0039] Figure 2 This is the overall framework diagram of the model proposed in this invention.

[0040] Figure 3 This is a diagram illustrating the two-stage dynamic optimization principle of cross-domain data fusion strategy.

[0041] Figure 4 This is a structural framework diagram of the dual spatial perception enhancement mechanism (DSPM). Detailed Implementation

[0042] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Figure 1 As shown, an end-to-end autonomous driving method with cross-domain data fusion and dual-space perception enhancement includes the following steps: A. Extracting multimodal features A1. Extracting visual features First, given a set to be generated In each camera view, since each front view image is associated with a different camera pose, therefore, The multi-frame, multi-view images from a single camera are represented as follows:

[0043] In the formula, , and They are the first The intrinsic parameters, extrinsic rotation matrix, and translation vector of each camera; Number of cameras; The number of historical timestamps during the vehicle's operation; , 3 and 3 represent the length, width, and number of three channels of the acquired image, respectively. In this embodiment of the invention, the camera view... Each RGB image is fed into a high-performance backbone network to generate multi-view features, represented as follows:

[0044] In the formula, This represents the number of channels.

[0045] A2. Extracting Text Features Language information is divided into several task-oriented subsets, with environmental perception and decision-making / planning subsets as typical examples, and a language information set is defined. The environmental perception subsequence and the decision-making and planning subsequence are respectively denoted as... and The hierarchical representation of language information is as follows:

[0046] in, The number of language-related tasks; The number of historical timestamps is the same as the timestamps of the image sequence. To transform discrete language symbols into continuous semantic features for model processing, word embedding is first performed on each language text information, mapping it to a low-dimensional dense vector; subsequently, instruction temporal encoding vectors are introduced. Ultimately, a unified dimension of language features is obtained. The formula is as follows:

[0047] In the formula, For the first Language-like modalities in the first The original linguistic information of the moment; and These represent the instruction sequence length and the text feature dimension, respectively. This is a word embedding function.

[0048] B. Constructing a cross-domain data fusion strategy The cross-domain data fusion strategy proposed in this invention divides the training process into two stages for dynamic optimization. The first stage is an active weight allocation stage based on dataset differences. This stage mainly calculates the static cross-domain redundancy based on the inherent differences in the feature space of different datasets, thereby establishing the initial sampling priority for various scenarios. The second stage is a delay adjustment stage based on loss feedback. This stage introduces the dynamic loss changes during the training process as a feedback signal to perform secondary fine-tuning of the weights for scenarios where learning stagnates. Through the synergistic effect of these two stages, balanced training for both conventional and long-tail scenarios is achieved.

[0049] B1. First Stage: Assigning Dynamic Weights Based on Dataset Differences First, define the training data set. To accurately measure the characteristic attributes of various scenarios, modal feature representation is adopted, that is, visual features are used. and text semantic features This indicates that visual features and text semantic features The weighted fusion yields:

[0050] In the formula, For scene feature vectors; For dataset The number of samples; For feature fusion weights. For dataset Visual features; For dataset Textual features. In embodiments of the present invention, .

[0051] To eliminate the size differences in different modal characteristics, Normalization is performed, and the normalized scene feature vector is calculated using the following formula. :

[0052] In the formula, It is an L2 norm.

[0053] Next, the cross-domain redundancy between different datasets is quantified for any two datasets. and The difference between the two is expressed as Further calculations on a single dataset Average difference with all other datasets The formula is as follows:

[0054] In the formula, The total number of data sets; For dataset and dataset Differences; This represents the average difference between datasets.

[0055] Then, based on the average difference Update sampling weights Every preset training step length The sampling weights are adjusted once based on the differences in the dataset. Based on this, the sampling weights are then... In the dataset at each training step size The weight iterative update function is expressed as:

[0056] In the formula, For the first Training dataset in rounds The sampling weights, which are only applied during training epochs. To update step size Update when the value is an integer multiple of the total value. This is the parameter for adjusting the weight. This represents the scene preference coefficient. In this embodiment of the invention, When it is a normal scenario When in edge scenes This is to ensure that edge data is fully sampled during training.

[0057] To suppress extreme value fluctuations, improve training stability, and ensure that the weights conform to the probability distribution, the updated weights are... After smoothing and normalization, it is represented as:

[0058]

[0059] In the formula, For the smoothed first A dataset in Weights for training step size; For smoothing terms; For the smoothed first A dataset in Weights for training step size; For the normalized first A dataset in Weights for training step size. In this embodiment of the invention... This is to prevent training instability caused by excessive fluctuations in weights due to differences.

[0060] B2. Second Stage: Dynamic Fine-tuning Based on Loss Feedback To avoid false triggering caused by fluctuations in single-round losses, a hysteresis adjustment mechanism based on loss feedback is introduced. Let the... Step in dataset The training loss is Then the first Wheel relative to the first The percentage decrease in wheel losses is defined as:

[0061] In the formula, The constraints ensure that only the decrease in losses is included in the statistics; This is the loss from the previous round. If it continues... Each training step size satisfies the loss reduction ratio. Less than the preset threshold Then determine When a learning stagnation state is reached, a loss feedback adjustment factor is introduced. , is represented as:

[0062] In the formula, To adjust the sensitivity coefficient; The global average loss is used to reflect the first... The overall difficulty level of the round training; This is the overfitting risk coefficient. If stagnant learning occurs on some datasets, then... If it does not stop, then Since the application of the adjustment factor causes the sum of the weight vectors to deviate from 1, it is mapped to the probability distribution space, ultimately yielding the first... Round sampling weights ,Right now In the embodiments of the present invention To control the magnitude of the impact of the loss on the weights.

[0063] C. Establish a dual-space perception enhancement module C1, 3D visual environment perception By utilizing depth uncertainty estimation based on Gaussian sputtering, multi-view image features are projected onto 3D space and rasterized to generate global features for a bird's-eye view, as shown in the following formula:

[0064] In the formula, For the first The image domain of a camera; For the first Splash characteristics of individual cameras; For the first The transparency parameters of each camera; and The first Pixels per camera The mean and covariance on the bird's-eye view plane. To achieve dynamic object feature extraction and static map feature extraction from the global features of the bird's-eye view, three learnable semantic query vectors are introduced. , and These query vectors serve as semantic anchors for vehicles, pedestrians, and static maps, respectively. They interact with the bird's-eye view features through a cross-attention mechanism, adaptively focusing on the corresponding semantic regions. The final output consists of three types of semantic features. And all three belong to space.

[0065] C2. Extract fine-grained features from the surrounding view local area. Calculate the surround view image using a multimodal pre-trained model based on contrastive learning. With language and text information Cosine similarity is used to generate an attention heatmap. Identify local maxima in the attention heatmap and calculate the adaptive cropping window size. The formula is as follows:

[0066] in, To expand the coefficient, avoid cropping the edge details of small targets; For the first The first camera The pixel area of ​​a cropped image; Minimum window size to ensure feature richness. Centered on the centroid, with dimensions as cropped image The image encoder, re-input into a contrastive learning-based multimodal pre-trained model, extracts fine-grained features. And semantic information in sub-images.

[0067] C3. Perform deep fusion of dual spatial features To address the issue of missing fine-grained information in the feature space of bird's-eye view images, this paper achieves deep fusion of panoramic cropping features and global features of the bird's-eye view image, utilizing multi-view feature alignment loss. The consistency between the constraint clipping feature and the global features of the bird's-eye view is determined by the following formula:

[0068] In the formula, This represents the total number of cut blocks; These are the projection features of the bird's-eye view features at the corresponding locations; This represents the mean square error.

[0069] C4. Establish a cross-spatial attention module Based on the completion of multi-view feature semantic alignment, the global features of the bird's-eye view are used as the query. The aligned cropping features generate keys respectively. AND value Design a cross-spatial attention module and build and The complementary fusion mechanism injects local cropped features into the global features of the bird's-eye view, generating enhanced bird's-eye view features. :

[0070] D. Perform end-to-end trajectory planning and reasoning Enhanced bird's-eye view features Global features Local clipping features and text features The input to the Visual Language Model (VLM) is as follows:

[0071] In the formula, The output set is a joint set that includes environmental perception interpretation and future trajectory planning. For learnable projectors; It is a text segmenter.

[0072] The basic principles, main features, and advantages of this invention have been described above. Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed.

Claims

1. An autonomous driving method with cross-domain data fusion and dual-space perception enhancement, characterized in that: Includes the following steps: A. Extracting multimodal features; B. Construct a cross-domain data fusion strategy; C. Establish a dual-space perception enhancement module; D. Perform end-to-end trajectory planning and reasoning.

2. The autonomous driving method with cross-domain data fusion and dual-space perception enhancement according to claim 1, characterized in that: The steps for extracting multimodal features described in step A are as follows: A1. Extracting visual features First, given a set to be generated In each camera view, since each front view image is associated with a different camera pose, therefore, The multi-frame, multi-view images from a single camera are represented as follows: In the formula, , and They are the first The intrinsic parameters, extrinsic rotation matrix, and translation vector of each camera; Number of cameras; The number of historical timestamps during the vehicle's operation; , 3 and represent the length, width, and number of channels of the acquired image, respectively; each RGB image is fed into the feature extraction network to generate multi-view features, represented as: In the formula, Number of channels; A2. Extracting Text Features Language information is divided into several task-oriented subsets, with environmental perception and decision-making / planning subsets as typical examples, and a language information set is defined. The environmental perception subsequence and the decision-making and planning subsequence are respectively denoted as... and The hierarchical representation of language information is as follows: in, The number of language-related tasks; The number of historical timestamps is the same as the timestamps of the image sequence. To transform discrete language symbols into continuous semantic features for model processing, word embedding is first performed on each language text information, mapping it to a low-dimensional dense vector; subsequently, instruction temporal encoding vectors are introduced. Ultimately, a unified dimension of language features is obtained. The formula is as follows: In the formula, For the first Language-like modalities in the first The original linguistic information of the moment; and These represent the instruction sequence length and the text feature dimension, respectively. This is a word embedding function.

3. The autonomous driving method with cross-domain data fusion and dual-space perception enhancement according to claim 1, characterized in that: The steps for constructing the cross-domain data fusion strategy described in step B are as follows: The cross-domain data fusion strategy divides the training process into two dynamic optimization phases, spanning the entire iterative training cycle of the end-to-end autonomous driving model. The first phase is an active weight allocation phase based on dataset differences. This phase calculates the static cross-domain redundancy between datasets based on the inherent differences in feature space across different datasets, thereby establishing the initial sampling priority for various scenarios. Based on this, training batches are constructed through weighted sampling and input into the end-to-end autonomous driving model for training and loss calculation. The second phase is a delay adjustment phase based on loss feedback. This phase introduces dynamic loss changes during training as feedback signals to perform secondary fine-tuning of weights for scenarios where learning stagnates. Through the synergistic effect of these two phases, balanced training for both conventional and long-tail scenarios is achieved. The specific steps are as follows: B1. First Stage: Assigning Dynamic Weights Based on Dataset Differences First, define the training data set. To accurately measure the characteristic attributes of various scenarios, modal feature representation is adopted, that is, visual features are used. and text semantic features This indicates that visual features and text semantic features The weighted fusion yields: In the formula, For scene feature vectors; For dataset The number of samples; For feature fusion weights; For dataset Visual features; For dataset Textual features; To eliminate the size differences in different modal characteristics, Normalization is performed, and the normalized scene feature vector is calculated using the following formula. : In the formula, It is an L2 norm; Next, the cross-domain redundancy between different datasets is quantified for any two datasets. and The difference between the two is expressed as Further calculations on a single dataset Average difference with all other datasets The formula is as follows: In the formula, The total number of data sets; For dataset and dataset Differences; The average difference between datasets; Then, based on the average difference Update sampling weights Every preset training step length The sampling weights are adjusted once based on the differences in the dataset. Based on this, the sampling weights are then... In the dataset at each training step size The weight iterative update function is expressed as: In the formula, For the first Training dataset in rounds The sampling weights, which are only applied during training epochs. To update step size Update when the value is an integer multiple of the total value. This is the parameter for adjusting the weight. The scene preference coefficient is used; to suppress extreme value fluctuations, improve training stability, and ensure that the weights satisfy the probability distribution, the updated weights are... After smoothing and normalization, it is represented as: In the formula, For the smoothed first A dataset in Weights for training step size; For smoothing terms; For the smoothed first A dataset in Weights for training step size; For the normalized first A dataset in Weights for training step size; Based on the above calculations, the normalized sampling weight vector The end-to-end autonomous driving model training iteration is performed, and the specific steps are as follows: B11, Data Sampling: In the... During step training, based on the sampling weight vector From the data set Randomly selected from the middle Training batch of samples; B12. Forward Propagation: The sampled multimodal data is input into the subsequent step CD to obtain a joint output set containing environmental perception interpretation and trajectory planning. ; B13. Loss Function Calculation: Define the total training loss. It includes environmental perception explanation loss. Multi-view feature alignment loss and trajectory planning loss Regarding the current time step Calculate the average loss value ; B14, Parameter Update: Utilizing The parameters are updated via backpropagation, and the specific loss for each dataset is recorded. As a feedback signal, it is transmitted to the second stage for dynamic fine-tuning of the weights; B2. Second Stage: Dynamic Fine-tuning Based on Loss Feedback To avoid false triggering caused by fluctuations in single-round losses, a hysteresis adjustment mechanism based on loss feedback is introduced; let the first... Step in dataset The training loss is Then the first Wheel relative to the first The percentage decrease in wheel losses is defined as: In the formula, The constraints ensure that only the decrease in losses is included in the statistics; This is for the losses from the previous round; If continuous Each training step size satisfies the loss reduction ratio. Less than the preset threshold Then determine When a learning stagnation state is reached, a loss feedback adjustment factor is introduced. , is represented as: In the formula, To adjust the sensitivity coefficient; The global average loss is used to reflect the first... The overall difficulty level of the round training; This represents the overfitting risk coefficient; if stagnant learning occurs on some datasets, then... If it does not stop, then Since the application of the adjustment factor causes the sum of the weight vectors to deviate from 1, it is mapped to the probability distribution space, ultimately yielding the first... Round sampling weights ,Right now .

4. The autonomous driving method with cross-domain data fusion and dual-space perception enhancement according to claim 1, characterized in that: The steps for establishing the dual-space perception enhancement module described in step C are as follows: C1, 3D visual environment perception By utilizing depth uncertainty estimation based on Gaussian sputtering, multi-view image features are projected onto 3D space and rasterized to generate global features for a bird's-eye view, as shown in the following formula: In the formula, For the first The image domain of a camera; For the first Splash characteristics of individual cameras; For the first The transparency parameters of each camera; and The first Pixels per camera The mean and covariance on the bird's-eye view plane; to achieve dynamic object feature extraction and static map feature extraction from the global features of the bird's-eye view, three learnable semantic query vectors are introduced. , and These serve as semantic anchors for vehicles, pedestrians, and static maps, respectively. These query vectors interact with the bird's-eye view features through a cross-attention mechanism, adaptively focusing on the corresponding semantic regions; ultimately outputting three types of semantic features. And all three belong to space; C2. Extract fine-grained features from the surrounding view local area. Calculate the surround view image using a multimodal pre-trained model based on contrastive learning. With language and text information Cosine similarity is used to generate an attention heatmap. Identify local maxima in the attention heatmap and calculate the adaptive cropping window size. The formula is as follows: in, To expand the coefficient, avoid cropping the edge details of small targets; For the first The first camera The pixel area of ​​a cropped image; To ensure the minimum window size for feature richness; Centered on the centroid, with dimensions as cropped image The image encoder, re-input into a contrastive learning-based multimodal pre-trained model, extracts fine-grained features. And semantic information in sub-images; C3. Perform deep fusion of dual spatial features To address the issue of missing fine-grained information in the feature space of bird's-eye view images, this paper achieves deep fusion of panoramic cropping features and global features of the bird's-eye view image, utilizing multi-view feature alignment loss. The consistency between the constraint clipping feature and the global features of the bird's-eye view is determined by the following formula: In the formula, This represents the total number of cut blocks; These are the projection features of the bird's-eye view features at the corresponding locations; Mean squared error; C4. Establish a cross-spatial attention module Based on the completion of multi-view feature semantic alignment, the global features of the bird's-eye view are used as the query. The aligned cropping features generate keys respectively. AND value Design a cross-spatial attention module and build and The complementary fusion mechanism injects local cropped features into the global features of the bird's-eye view, generating enhanced bird's-eye view features. : 。 5. The autonomous driving method with cross-domain data fusion and dual-space perception enhancement according to claim 1, characterized in that: The steps for end-to-end trajectory planning and reasoning described in step D are as follows: Enhanced bird's-eye view features Global features Local clipping features and text features The input formula for the Visual Language Model (VLM) is as follows: In the formula, The output set is a joint set that includes environmental perception interpretation and future trajectory planning. For learnable projectors; It is a text segmenter.

Images