Intelligent vehicle-oriented three-dimensional target detection parameter efficient migration fine tuning method

Abstract

The invention relates to the technical field of intelligent vehicles, and discloses an intelligent vehicle-oriented three-dimensional target detection parameter efficient migration fine tuning method, which comprises the following steps of: acquiring and preprocessing multi-modal sensor data of a vehicle to obtain multi-modal data, inputting a parameter frozen multi-modal three-dimensional target detection backbone network, and carrying out multi-modal three-dimensional target detection on the multi-modal three-dimensional target detection backbone network; obtaining bird's-eye view features of each mode and unified bird's-eye view fusion features; inserting an adapter module, performing fine adjustment on the aerial view features of each mode and the unified aerial view fusion features, and constructing a representation side migration subspace; applying a low-rank re-parameterization module, performing incremental updating on the weight of the three-dimensional target detection head, and constructing a discrimination side migration subspace; basic detection loss and consistency constraint loss are calculated, and a total loss function is constructed; and updating the adapter module and the low-rank re-parameterization module to obtain a three-dimensional target detection model after migration and fine tuning. And the computing resource consumption and the adaptive training period of the vehicle-mounted terminal are reduced.

Description

An efficient transfer fine-tuning method for 3D target detection parameters for intelligent vehicles Technical Field

[0001] This invention relates to the field of intelligent vehicle technology, specifically to an efficient method for fine-tuning the transfer parameters of three-dimensional target detection for intelligent vehicles. Background Technology

[0002] With the evolution of advanced autonomous driving technology, environmental perception systems have evolved from single-sensor systems to multimodal fusion. Mainstream intelligent vehicle perception systems are typically equipped with multiple sensors such as cameras, LiDAR, and 4D millimeter-wave radar. By constructing a unified bird's-eye view (BEV) or voxel feature space, they utilize deep neural networks to achieve 3D detection of vehicles, pedestrians, and traffic facilities. Current perception models typically require large-scale, full-parameter training using massive amounts of labeled data under specific urban road conditions, weather conditions, and fixed sensor configurations to ensure high accuracy and robustness in detection. Existing related technologies mainly focus on designing more complex feature extraction and fusion network structures to improve the perception performance of single-trained models in specific scenarios.

[0003] However, in actual large-scale deployments, the vehicle's operating environment (such as urban style and climate change) and sensor configuration (such as replacement with new models and differences in installation pose) often change. Existing technologies face the following technical challenges when dealing with such cross-domain migration problems: First, traditional methods usually require full parameter retraining or large-scale fine-tuning of a large multimodal detection network, which not only consumes huge computational resources but also has a long adaptation cycle, making it difficult to meet the needs of rapid iteration of on-board computing platforms. Second, most existing parameter-efficient fine-tuning methods are designed for natural language or 2D image tasks. When directly applied to multimodal bird's-eye view networks with complex spatial hierarchical structures, they are prone to disrupting the established geometric and semantic alignment relationships between modalities, resulting in low migration efficiency. In addition, existing fine-tuning strategies often lack stability constraints for 3D geometry (position, size) and motion physical quantities (velocity, orientation), and do not fully consider the differences in the physical reliability of sensors of different modalities under different weather conditions. This leads to problems such as geometric estimation divergence, motion state prediction drift, and noise amplification in weak evidence scenarios during small-sample fine-tuning. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: an efficient transfer fine-tuning method for 3D target detection parameters in intelligent vehicles, comprising: acquiring and preprocessing multimodal sensor data of the vehicle to obtain multimodal data in a unified coordinate system; inputting the multimodal data into a multimodal 3D target detection backbone network with frozen parameters and performing forward inference to obtain bird's-eye view features of each modality and unified bird's-eye view fusion features; inserting adapter modules into the paths of each modal bird's-eye view feature and unified bird's-eye view fusion features to fine-tune the bird's-eye view features of each modality and the unified bird's-eye view fusion features, constructing a representation-side transfer subspace; applying a low-rank parameterization module to a preset layer of the 3D target detection head to incrementally update the weights of the 3D target detection head, constructing a discriminative-side transfer subspace; calculating the basic detection loss and consistency constraint loss based on the representation-side transfer subspace and the discriminative-side transfer subspace, and constructing a total loss function; updating the adapter module and the low-rank parameterization module based on the total loss function to obtain a 3D target detection model after transfer fine-tuning.

[0006] As a preferred embodiment of the efficient migration and fine-tuning method for 3D target detection parameters for intelligent vehicles described in this invention, the fine-tuning of the modal bird's-eye view features and the unified bird's-eye view fusion features includes: inserting modality-specific adapters into the paths of the modality-specific bird's-eye view features, and inserting a shared adapter into the path of the unified bird's-eye view fusion features; using the dimensionality reduction units in the modality-specific adapters and the shared adapters to compress the number of channels of the input features to a preset number of channels; applying nonlinear activation to the compressed features through a nonlinear transformation unit to obtain activated features; restoring the number of channels of the activated features to the original number of channels through an up-dimensionality unit to obtain up-dimensional features; and using a residual connection unit to add the up-dimensional features to the input features element-wise to obtain the fine-tuned bird's-eye view features.

[0007] As a preferred embodiment of the efficient migration and fine-tuning method for 3D target detection parameters for intelligent vehicles described in this invention, the construction of the discriminative side migration subspace includes: determining a preset layer in the 3D target detection head; the preset layer includes a 3D bounding box branch layer, an orientation and velocity branch layer, and an occupancy attribute branch layer; generating a first low-rank matrix and a second low-rank matrix corresponding to the original weight matrix in the preset layer; calculating incremental weights based on the first low-rank matrix and the second low-rank matrix, and combining the incremental weights with the frozen original weight matrix to obtain an updated weight matrix; configuring corresponding rank values ​​for each preset layer; wherein the rank values ​​configured for the 3D bounding box branch layer and the orientation and velocity branch layer are greater than the rank value of the occupancy attribute branch layer.

[0008] As a preferred embodiment of the efficient migration and fine-tuning method for 3D target detection parameters for intelligent vehicles described in this invention, the consistency constraint loss includes geometric consistency loss, velocity consistency loss, and cross-modal consistency loss. Calculating the consistency constraint loss includes: obtaining the geometric reference prediction result output by the parameter-frozen multimodal 3D target detection backbone network; obtaining the current geometric prediction result output after updating through the representation-side migration subspace and the discriminative-side migration subspace; calculating the geometric consistency loss based on the geometric reference prediction result and the current geometric prediction result; obtaining target velocity information from the multimodal sensor data as a velocity reference value; obtaining the target velocity prediction value output by the 3D target detection head; calculating the velocity consistency loss based on the velocity reference value and the target velocity prediction value; obtaining the occupancy probability inferred from each modality in a unified bird's-eye view grid; calculating the weighted average occupancy probability based on the reliability weight of each modality; and calculating the cross-modal consistency loss based on the occupancy probability of each modality and the weighted average occupancy probability.

[0009] As a preferred embodiment of the efficient migration and fine-tuning method for 3D target detection parameters for intelligent vehicles described in this invention, the following steps are taken: obtaining the reliability weights of each modality includes: for each modality, collecting environmental state information and sensor self-diagnostic information of the current scene, and extracting feature statistics for each modality to construct a reliability assessment input vector; wherein the environmental state information includes weather conditions and illumination information; inputting the reliability assessment input vector into a reliability assessment network to obtain a reliability score for each modality; and normalizing the reliability scores of all modalities to obtain the reliability weights of each modality.

[0010] As a preferred embodiment of the efficient migration fine-tuning method for 3D target detection parameters for intelligent vehicles described in this invention, the reliability weight used to adjust the output of the adapter module and the consistency constraint loss includes: for each modality, calculating the adapter output features by weighting the reliability weight of each modality with the output features of the modality-specific adapter corresponding to the modality; fusing the adapter output features of each modality to make the modality with high reliability generate a larger feature contribution in the representation-side migration subspace; weighting the contributions of each modality in the geometric consistency loss, the velocity consistency loss, and the cross-modal consistency loss according to the reliability weight, so that the modality with high reliability occupies a larger proportion in loss optimization; when the reliability weight of a modality is less than a preset gating threshold, adjusting the output features of the modality-specific adapter using an attenuation coefficient to reduce the contribution ratio of the modality in feature fusion and loss calculation.

[0011] As a preferred embodiment of the efficient transfer and fine-tuning method for 3D target detection parameters for intelligent vehicles described in this invention, the method involves: inserting modality-specific adapters into the paths of the features in each modality bird's-eye view, and inserting a shared adapter into the path of the unified bird's-eye view fusion features. This includes: analyzing the feature layers in the paths of the features in each modality bird's-eye view, selecting feature layers with spatial resolution less than a preset resolution threshold and channel number greater than a preset channel number threshold, and using these as insertion positions for the modality-specific adapters; analyzing the feature layers in the paths of the unified bird's-eye view fusion features, selecting feature layers with spatial resolution less than a preset resolution threshold and channel number greater than a preset channel number threshold, and using these as insertion positions for the shared adapters; and inserting the modality-specific adapter and the shared adapter at the insertion positions.

[0012] A highly efficient parameter transfer and fine-tuning system for 3D target detection in intelligent vehicles includes: a target detection backbone network module that acquires and preprocesses multimodal sensor data from the vehicle to obtain multimodal data in a unified coordinate system; inputting the multimodal data into a parameter-frozen multimodal 3D target detection backbone network and performing forward inference to obtain bird's-eye view features for each modality and a unified bird's-eye view fusion feature; and a representation-side transfer module that inserts adapter modules into the paths of the bird's-eye view features for each modality and the unified bird's-eye view fusion feature to adjust the parameters of the bird's-eye view features for each modality and the unified bird's-eye view fusion feature. The features are fine-tuned to construct a representation-side transfer subspace; the discriminant-side transfer module applies a low-rank parameterization module to a preset layer of the 3D target detection head, incrementally updating the weights of the 3D target detection head to construct a discriminant-side transfer subspace; the constraint construction module calculates the basic detection loss and consistency constraint loss based on the representation-side transfer subspace and the discriminant-side transfer subspace and constructs a total loss function; the transfer fine-tuning module updates the adapter module and the low-rank parameterization module based on the total loss function to obtain the transfer-fine-tuned 3D target detection model.

[0013] A computer device includes: a memory and a processor; the memory stores a computer program, wherein: when the processor executes the computer program, it implements the steps of the method described in any one of the present invention.

[0014] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the present invention.

[0015] The beneficial effects of this invention are as follows: By freezing the multimodal backbone network and constructing dual transfer subspaces for the representation and discriminative sides, this invention enables rapid model adaptation across different cities, weather conditions, and sensor configurations with only a few parameter updates, reducing computational resource consumption and adaptation training cycle on the vehicle side. By inserting adapters into the feature paths of the bird's-eye view to construct the representation subspace and applying differentiated low-rank parameterization to the preset layer of the detection head to construct the discriminative subspace, the feature domain offset repair and discriminative boundary adjustment are effectively decoupled. While maintaining the stability of the backbone network's geometric representation, the discriminative ability for 3D bounding boxes and velocities is prioritized. By introducing geometric consistency, velocity consistency, and cross-modal consistency constraints, a multi-dimensional physical supervision mechanism is constructed, effectively suppressing the drift and divergence of 3D position, size, and velocity estimation during few-sample fine-tuning, ensuring the accuracy of geometric and motion estimation after transfer. By using modal reliability weights to gate and weight the adapter output and loss function, adaptive perception of different environmental conditions can be achieved. This can automatically enhance the contribution of high-reliability modes and suppress noise interference from degenerate modes, thereby improving the robustness and reliability of multimodal perception in intelligent vehicles. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 is an overall flowchart of a method for efficient migration and fine-tuning of three-dimensional target detection parameters for intelligent vehicles provided by an embodiment of the present invention. Detailed Implementation

[0018] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0019] Example 1, referring to Figure 1, is an embodiment of the present invention, providing an efficient transfer and fine-tuning method for 3D target detection parameters for intelligent vehicles, including: S1: acquiring multimodal sensor data of the vehicle and preprocessing it to obtain multimodal data in a unified coordinate system, inputting the multimodal data into a multimodal 3D target detection backbone network with frozen parameters and performing forward inference to obtain bird's-eye view features of each modality and unified bird's-eye view fusion features.

[0020] In this embodiment, the implementation of the efficient transfer and fine-tuning method for multimodal 3D target detection parameters of intelligent vehicles relies on accurate perception of the vehicle's surrounding environment. Through multimodal sensor acquisition, data preprocessing and calibration, and forward inference of the backbone network, complete feature information required for the subsequent construction of the representation-side transfer subspace and the discriminant-side transfer subspace is obtained. The obtained feature information includes not only the bird's-eye view feature representation of each modality, but also the unified bird's-eye view fusion feature. These feature information together constitute the input basis of the efficient transfer and fine-tuning method for parameters.

[0021] In multimodal 3D target detection systems for intelligent vehicles, to improve reliability in weak evidence scenarios such as long distances, occlusion, and rain / fog, the mainstream technical approach has gradually expanded from single-sensor LiDAR perception to multi-sensor fusion such as camera-LiDAR-millimeter-wave radar, achieving multimodal 3D perception through unified representations such as bird's-eye view or voxels. However, existing multimodal 3D target detection networks typically have deep backbones, multimodal fusion modules, and multi-branch detection heads, resulting in a large number of parameters and high structural coupling. When the scene or sensor configuration changes, it is often necessary to retrain most of the backbone and fusion layers with full parameters or perform large-scale fine-tuning, leading to high computational overhead and long adaptation cycles. Therefore, this invention adopts a parameter freezing strategy when acquiring multimodal data and performing backbone network inference, laying the foundation for subsequent updates to only a small number of migration subspace parameters.

[0022] Specifically, the process of acquiring and preprocessing the multimodal sensor data of the vehicle to obtain multimodal data in a unified coordinate system includes the following steps: First, during the vehicle's operation, multimodal data at the current moment and adjacent moments are collected using cameras, lidar, and 4D millimeter-wave radar.

[0023] It should be noted that the multimodal data specifically includes: camera images refer to image frames acquired by multiple surround-view or forward-view cameras at the current and adjacent times. The images can be RGB or grayscale images, and each frame includes a timestamp, camera number, and other information. Furthermore, camera images may also include imaging parameters such as exposure time, gain, and shutter mode for subsequent brightness and signal-to-noise ratio evaluation. LiDAR point clouds refer to the original three-dimensional point clouds at the current and adjacent times, with each point containing at least three-dimensional coordinates. The data includes reflection intensity, and if necessary, line number, local time offset. The entire point cloud frame includes scan start and end times and radar number. 4D millimeter-wave radar data is in target list or radar grid format. When in target list format, each radar target includes at least: range, azimuth, elevation or altitude correlation, radial velocity (Doppler velocity), and radar cross section. When in grid format, each grid cell includes echo intensity, presence probability, radial velocity statistics, etc. In an optional embodiment, multimodal data may also include vehicle motion and attitude information, i.e., vehicle pose, velocity, and acceleration obtained from inertial navigation, GPS, and wheel speed sensors, used to unify the various modal data into the vehicle coordinate system or world coordinate system.

[0024] Secondly, the data preprocessing and calibration module is used to perform time synchronization, coordinate system transformation and basic filtering on the data of each mode, so as to unify the original data into the vehicle coordinate system or the world coordinate system and obtain multimodal data in a unified coordinate system.

[0025] Specifically, the time synchronization refers to the need to perform time alignment processing on the data of each modality to ensure the consistency of the data in time, since there are differences in the sampling frequency and triggering time of the camera, lidar and 4D millimeter wave radar.

[0026] The coordinate system transformation refers to the fact that each modal sensor has its own independent sensor coordinate system, and the data of each modality needs to be uniformly transformed into the vehicle coordinate system.

[0027] The basic filtering refers to filtering the data of each modality to remove noise and invalid data.

[0028] Specifically, the process of freezing the parameters of the multimodal data input into the multimodal 3D target detection backbone network and performing forward inference to obtain the bird's-eye view features of each modality and the unified bird's-eye view fusion features includes the following steps: inputting pre-processed camera images, LiDAR point clouds, and 4D millimeter-wave radar data into the pre-trained multimodal 3D target detection backbone network; the multimodal 3D target detection backbone network includes sub-networks for extracting features of each modality and a sub-network for fusing multimodal bird's-eye view features; wherein, the parameters of the multimodal 3D target detection backbone network remain frozen and do not participate in subsequent fine-tuning, but are only used to generate the bird's-eye view feature representations of each modality, the unified bird's-eye view fusion features, and the initial detection results generated by the basic detection head.

[0029] It should be noted that, since multimodal 3D target detection backbone networks typically have a large number of parameters and high structural coupling, and the network parameters are highly coupled between multimodal paths, geometric representations and multimodal alignment are fixed in a unified backbone and fusion structure. If full parameter fine-tuning is performed on the backbone network, it requires long-term training and a large amount of computing resources, which is difficult to implement under on-board computing power conditions. By freezing the backbone network parameters, the stability of the learned geometric and temporal representations can be maintained, avoiding the destruction of multimodal alignment relationships. At the same time, the fine-tuning range is limited to the adapter modules and low-rank parameterization modules added in subsequent steps, thereby reducing the computational overhead of the fine-tuning process.

[0030] Each modality feature extraction subnetwork is used to extract features from camera images, LiDAR point clouds, and 4D millimeter-wave radar data to obtain bird's-eye view feature representations for each modality. Specifically, this includes: performing convolutional feature extraction on camera images and mapping the features to a bird's-eye view grid through a geometric projection or viewpoint transformation module to obtain camera bird's-eye view features; performing voxelization or sparse convolution on LiDAR point clouds to obtain LiDAR bird's-eye view features; and encoding radar points or radar grids in 4D millimeter-wave radar data to generate radar bird's-eye view features.

[0031] Furthermore, the multimodal bird's-eye view feature fusion subnetwork fuses the bird's-eye view features of each modality under a unified bird's-eye view coordinate system to obtain a unified bird's-eye view fusion feature; specifically, the camera bird's-eye view features, lidar bird's-eye view features, and radar bird's-eye view features are fused according to a preset method; the preset method includes weighted fusion, stitching and convolution, etc.

[0032] It should be noted that the unified bird's-eye view fusion feature integrates complementary information from various modalities, taking into account the semantic expression capabilities of the camera, the geometric accuracy of the lidar, and the robustness of the millimeter-wave radar in adverse weather conditions. This can improve the reliability of perception in weak evidence scenarios such as long distances, obstructions, rain, and fog.

[0033] Furthermore, through the forward inference of the multimodal 3D target detection backbone network, the bird's-eye view feature representations of each modality, the unified bird's-eye view fusion features, and the initial detection results generated by the basic detection head are obtained; wherein, the bird's-eye view feature representations of each modality include camera bird's-eye view features, lidar bird's-eye view features, and radar bird's-eye view features.

[0034] It should be noted that, addressing the issue of high adaptation costs and large-scale parameter fine-tuning required for existing multimodal 3D object detection networks when adapting to cross-scene and cross-sensor configurations, this paper proposes a solution. By keeping the parameters of the multimodal 3D object detection backbone network frozen, stable bird's-eye view features for each modality and unified bird's-eye view fusion features are obtained, while avoiding full parameter retraining of the deep backbone and fusion structure. Since the number of newly added trainable parameters is less than the number of complete network parameters, and the training rounds can be shortened, adaptation can be completed by fine-tuning a small number of parameters in a short time when migrating between different cities and different sensor configurations. This reduces the computational resources and time costs of cross-scene and cross-sensor configuration adaptation, making it possible to achieve fast and stable cross-scene and cross-sensor configuration migration under vehicle computing power and limited annotation conditions.

[0035] S2: Insert adapter modules into the paths of each modal bird's-eye view feature and the unified bird's-eye view fusion feature to fine-tune the each modal bird's-eye view feature and the unified bird's-eye view fusion feature, and construct the representation-side migration subspace.

[0036] Since existing efficient parameter fine-tuning methods are mostly designed for single-modality or two-dimensional vision tasks, directly inserting general adapters or low-rank update structures uniformly into multimodal bird's-eye view networks can easily disrupt the learned geometric and temporal representations, making it difficult to balance the spatial hierarchy and fusion stability of each modality's features. This embodiment starts from the network function and selectively inserts adapter modules into the multimodal bird's-eye view path to absorb domain offsets without excessively disturbing the overall geometric structure. This avoids the drawback of indiscriminate insertion of general efficient parameter fine-tuning methods into multimodal bird's-eye view networks, allowing limited parameters to be used in the parts most sensitive to transfer effects.

[0037] Specifically, the step of inserting the adapter module into the paths of each modal bird's-eye view feature and the unified bird's-eye view fusion feature includes the following steps: inserting modality-specific adapters into the paths of each modality bird's-eye view feature and inserting a shared adapter into the path of the unified bird's-eye view fusion feature, including: analyzing the feature layers in the paths of each modality bird's-eye view feature and selecting feature layers with spatial resolution less than a preset resolution threshold and channel number greater than a preset channel number threshold as the insertion positions of the modality-specific adapters; analyzing the feature layers in the paths of the unified bird's-eye view fusion feature and selecting feature layers with spatial resolution less than a preset resolution threshold and channel number greater than a preset channel number threshold as the insertion positions of the shared adapters; inserting the modality-specific adapter and the shared adapter at the insertion positions; wherein, the modality-specific adapter is used to absorb modality-specific representation offsets caused by changes in sensor model and installation pose; the shared adapter is used to absorb cross-modality common representation offsets caused by scene changes; through the combined configuration of the modality-specific adapter and the shared adapter, modality-specific offsets and cross-modality common offsets can be processed simultaneously under the premise of limited parameter budget.

[0038] For example, in each modal bird's-eye view feature path and unified bird's-eye view fusion path, a set of low-resolution, high-channel-count feature layers are selected as the insertion positions of the adapter. Specifically, the low-resolution feature layers are defined as follows: Let the size of the highest-resolution feature map in the unified bird's-eye view space be... If the spatial dimensions of a certain bird's-eye view feature layer are Then when and It is determined to be a low-resolution feature layer, or when and When this occurs, it is determined to be a lower resolution layer. The definition of the high channel number feature layer is as follows: Let the maximum number of channels in all feature layers of the multimodal bird's-eye view backbone network be... And set a channel threshold. (In this embodiment, the value is 128 or 256). If the number of channels in a certain layer is... When satisfied or When the time is right, the corresponding layer is determined to be a high-channel-count feature layer.

[0039] Furthermore, the layer selection principle is as follows: in each modal bird's-eye view feature path and unified bird's-eye view fusion path, priority is given to selecting 1 to 3 feature layers that simultaneously satisfy the conditions of lower spatial resolution and higher channel number to insert into the adapter module. For example, when the highest bird's-eye view resolution is... And when there are 4 scales, it is possible to only use a scale with a size of approximately , Furthermore, adapters are inserted on layers with 128 or 256 channels to balance rich semantic information with controllable computational overhead.

[0040] In this embodiment, the representation-side migration subspace is implemented by inserting a bottleneck adapter module. For any selected bird's-eye view feature layer, the bottleneck adapter module includes an input interface, a dimensionality reduction unit, a nonlinear transformation unit, a dimensionality increase unit, and a residual connection unit connected in sequence. The input interface is used to receive feature maps from the corresponding modality or the unified bird's-eye view path.

[0041] Specifically, the fine-tuning of the bird's-eye view features of each modality and the unified bird's-eye view fusion features includes the following steps: Using the dimensionality reduction units in the modality-specific adapter and the shared adapter, the number of channels of the input features is compressed to a preset number. A nonlinear activation is applied to the compressed features through a nonlinear transformation unit to obtain activated features. The number of channels of the activated features is restored to the original number through an upscaling unit; the upscaling unit achieves channel restoration through pointwise convolution or linear transformation. The upscaling features are element-wise added to the input features using a residual connection unit to obtain the fine-tuned bird's-eye view features.

[0042] Furthermore, compressing the number of input feature channels to a preset number using the modality-specific adapter and the dimensionality reduction unit in the shared adapter includes: the dimensionality reduction unit compressing the number of feature channels to a preset ratio of the original number of channels through pointwise convolution or linear transformation, wherein the number of input feature channels of the adapter is 1. The number of channels after dimensionality reduction is The specific formula for the preset ratio is as follows: ;in, This is the channel compression ratio coefficient; This is the lower bound for the number of channels after dimensionality reduction, used to avoid excessive compression that could lead to a loss of expressive power; Rounding is performed to the nearest integer.

[0043] For example, typical values ​​for the above-mentioned preset ratio formula are as follows: When , , hour, ;when , hour, Furthermore, through a pre-defined design ratio, the number of new parameters added to the adapter is approximately... Compared to the main network weights, it accounts for only a small proportion, enabling efficient fine-tuning of parameters.

[0044] It should be noted that, in order to address the problem that existing general parameter fine-tuning methods are difficult to apply directly to multimodal bird's-eye view structures, a representation-side migration subspace is constructed, and bottleneck adapter modules are inserted only on key feature layers with low resolution and high channel count to absorb domain offset without excessively perturbing the overall geometry.

[0045] S3: Apply a low-rank parameterization module to the preset layer of the 3D target detection head to incrementally update the weights of the 3D target detection head and construct a discriminative side migration subspace.

[0046] After obtaining the representation-side transfer subspace constructed in S2, this embodiment also needs to construct the discrimination-side transfer subspace to refine the target classification and regression discrimination boundary. This is achieved by applying a low-rank heavy parameterization module to the preset layer of the 3D target detection head, while keeping the original detection head backbone weights frozen, by using only a small number of low-rank incremental parameters to refine the discrimination capabilities of the regression branches such as target category, 3D bounding box, and velocity.

[0047] Specifically, applying a low-rank parameterization module to a preset layer of the 3D target detection head includes the following steps: determining a preset layer in the 3D target detection head; the preset layer includes a 3D bounding box branch layer, an orientation and velocity branch layer, and an occupancy attribute branch layer; wherein, the 3D bounding box branch layer is used to output the category, 3D center, and size of each candidate target; the orientation and velocity branch layer is used to output the target orientation angle and velocity vector; and the occupancy attribute branch layer is used to output the scene occupancy map and other semantic attributes.

[0048] For the original weight matrix in the preset layer, a first low-rank matrix and a second low-rank matrix corresponding to the original weight matrix are generated. Incremental weights are calculated based on the first and second low-rank matrices. These incremental weights are then combined with the frozen original weight matrix to obtain an updated weight matrix. The specific process is as follows: For some weights in the preset layer, trainable incremental parameters are introduced through low-rank reparameterization. Specifically, for weight matrices requiring fine-tuning, the incremental part of the weight matrix is ​​represented as the product of two low-dimensional matrices, and during training, only the parameters of these two low-dimensional matrices are updated, while the original weight matrix remains frozen.

[0049] For any output branch (such as the 3D bounding box branch, orientation and velocity branch, or occupancy attribute branch), the original fully connected layer weight matrix can be written as: Where W is the original weight that is frozen; For output dimensions; To detect the number of feature channels in the head.

[0050] Under the low-rank parameterization strategy, the fine-tuned weight matrix is ​​as follows: Where W is the original weight that is frozen; , For rank The low-rank increment matrix, only and Updated during fine-tuning; This represents the rank of the corresponding output branch.

[0051] Furthermore, corresponding rank values ​​are configured for each of the preset layers; wherein the rank values ​​configured for the 3D bounding box branch layer and the orientation and velocity branch layer are greater than the rank value of the occupancy attribute branch layer.

[0052] For example, assuming the detection head has three types of output branches, the rank values ​​for different output branches are configured as follows: For the 3D bounding box branch, the output quantities include geometric quantities such as center and size, which have the greatest impact on overall performance; the rank value is taken. This ensures strong geometric transfer capabilities; if , The number of new parameters is approximately Compared to a full update Smaller. For the orientation and velocity branches, responsible for target orientation and velocity estimation, which are crucial for safety in dynamic scenes; rank is taken. Slightly larger than or equal to the rank value Equivalent, for example This ensures that the ability to determine orientation and velocity has sufficient flexibility to adapt to new scenarios. For the occupancy attribute branch, which is mainly used to assist in scene understanding, its importance is slightly lower than geometry and velocity; when the parameter budget is limited, low-rank parameterization can be avoided, i.e., keeping this branch completely frozen; or a lower rank can be used, such as... Only limited adjustments were made.

[0053] In addition, in the total parameter budget Given the given conditions, prioritize assigning larger ranks to the 3D bounding box branch and the orientation and velocity branches, such as the rank value. , This ensures that key physical quantities have stronger expressive and adaptability capabilities. For occupancy attribute branches, only small ranks are allocated when the budget allows, or no low-rank parameters are allocated, so that limited parameter resources are concentrated on improving key outputs such as 3D position, scale, orientation, and velocity. Finally, through a low-rank configuration strategy based on the branch level, the ability to distinguish key physical quantities such as geometry and velocity is prioritized to improve under the condition that the overall parameter increment is limited.

[0054] It should be noted that, addressing the issue that existing general-purpose parameter-efficient fine-tuning methods are difficult to directly apply to multimodal bird's-eye view structures, a discriminant-side transfer subspace is constructed. Low-rank heavy parameterization is applied only to a few layers at the end of the detection head, focusing on adjusting the classification and regression discrimination boundaries. Since the representation-side transfer subspace and the discriminant-side transfer subspace are respectively positioned and functionally adjusted for representation domain shift and discrimination boundary, the drawback of indiscriminate insertion of general-purpose parameter-efficient fine-tuning methods in multimodal bird's-eye view networks is avoided. This allows limited parameters to be used in the parts most sensitive to the transfer effect, thereby improving the applicability and effectiveness of parameter-efficient fine-tuning methods in multimodal bird's-eye view 3D detection scenarios.

[0055] S4: Based on the representation-side migration subspace and the discrimination-side migration subspace, calculate the basic detection loss and consistency constraint loss, and construct the total loss function.

[0056] Furthermore, after obtaining the representation-side transfer subspace and the discriminant-side transfer subspace constructed in S2 and S3, it is necessary to construct a total loss function to guide parameter updates. The total loss function is the core optimization objective of the present invention for efficient parameter transfer fine-tuning. By weighting and combining the basic detection loss with various consistency constraint losses, the three-dimensional geometry and motion estimation are stabilized while ensuring detection performance.

[0057] Specifically, calculating the basic detection loss includes: for the current batch of training samples, calculating the basic loss function for multimodal 3D object detection based on the fine-tuned multimodal bird's-eye view features and the detection head output. The basic detection loss typically includes target classification loss, 3D bounding box position and size regression loss, and orientation regression loss.

[0058] In this embodiment, the consistency constraint loss includes geometric consistency loss, velocity consistency loss, and cross-modal consistency loss.

[0059] Specifically, calculating the consistency constraint loss includes the following steps: obtaining the geometric reference prediction result output by the parameter-frozen multimodal 3D target detection backbone network; obtaining the current geometric prediction result output after being updated by the representation-side migration subspace and the discriminant-side migration subspace; and calculating the geometric consistency loss based on the geometric reference prediction result and the current geometric prediction result; specifically including the following steps: for the... The goal is to freeze the reference center of the backbone network output. The new center of the network output after fine-tuning is The reference size vector is The new dimension vector after fine-tuning is ,in, , and These are the three-dimensional coordinates of the target center; , and These are the target's width, length, and height, respectively. For reference only; This is the adjusted predicted value.

[0060] Secondly, a location consistency loss is constructed based on the differences between the reference center and the new center; let the number of targets participating in the geometric constraints be... , No. The weights of each objective are: Then the position consistency loss The specific formula is as follows: ;in, This is a commonly used smoothing L1 loss.

[0061] A size consistency loss is constructed based on the difference between the reference size and the new size. The specific formula is as follows: The positional consistency loss and the dimensional consistency loss are weighted and combined to obtain the total geometric consistency loss, as shown in the following formula: ;in, This represents the total loss for geometric consistency. , These are the weighting coefficients.

[0062] It should be noted that the new center of the network output after constraint fine-tuning New dimension vector after fine-tuning Do not deviate from the reference center of the frozen backbone network output. Reference size vector Even with only a few adapter and low-rank parameters updated, the abnormal drift of the target's 3D position and scale during fine-tuning in the new scene can be suppressed, maintaining the stability of the geometric estimate.

[0063] Further, the target velocity information in the multimodal sensor data is obtained as a velocity reference value, the target velocity prediction value output by the three-dimensional target detection head is obtained, and the velocity consistency loss is calculated based on the velocity reference value and the target velocity prediction value. Specifically, this includes the following steps: For the first... The target has a reference plane velocity obtained from the 4D radar. The planar velocity predicted by the 3D detection head is The orientation angle predicted by the detection head is ,in and Let be the velocity components of the target in the planar coordinate system, and let be the number of targets participating in the velocity constraint. The weight is The velocity amplitude consistency loss is obtained by calculating the deviation between the predicted velocity and the reference velocity and then performing a weighted average. The specific formula is as follows: ;in, This is the commonly used smoothing L1 loss; For reference only.

[0064] Furthermore, a velocity direction and orientation consistency loss is constructed, including the following steps: First, the unit vector corresponding to the orientation angle is calculated based on the orientation angle predicted by the detection head. : Secondly, calculate the unit vector of the velocity direction based on the predicted velocity. : ;in, To prevent tiny constants with a denominator of zero.

[0065] An orientation consistency loss is constructed by calculating the deviation between the dot product of the unit vector in the velocity direction and the unit vector in the orientation angle. : The velocity amplitude consistency loss and velocity direction consistency loss are weighted and combined to obtain the comprehensive velocity consistency loss, as shown in the following formula: ;in, This is to account for the overall speed consistency loss; These are the weighting coefficients.

[0066] It should be noted that the overall speed consistency loss The constrained predicted velocity vector is numerically close to the reference velocity given by the 4D radar, reducing velocity amplitude error; this also addresses the overall velocity consistency loss. By ensuring that the direction of the constrained velocity is consistent with or close to the predicted orientation, the physical inconsistency between the vehicle's orientation and the direction of motion is effectively suppressed. The combination of these two measures ensures that the velocity-orientation related errors are suppressed synchronously during the fine-tuning process, thus stabilizing the target motion estimation.

[0067] Furthermore, the occupancy probabilities inferred from each modality are obtained in the unified bird's-eye view grid. A weighted average occupancy probability is calculated based on the reliability weights of each modality. Cross-modal consistency loss is calculated based on the occupancy probabilities of each modality and the weighted average occupancy probability. Specifically, this includes the following steps: On the unified bird's-eye view grid, for the first... There are [number] grid cells, and the probability of the camera's bird's-eye view being occupied is [percentage]. The probability of occupancy of the LiDAR bird's-eye view is The occupancy probability of the 4D radar bird's-eye view is The corresponding modal reliability weight is The set of meshes participating in the constraints is Its size is The weighted average occupancy probability of each grid cell is calculated using modal reliability weights. The specific formula is as follows: Calculate the deviation between the occupancy probability of each mode and the weighted average occupancy probability, and construct the cross-modal consistency loss. The specific formula is as follows: ;in, For cross-modal consistency loss; The set of meshes participating in the constraints.

[0068] It should be noted that when a certain mode produces a high confidence error that deviates significantly from other modes in a certain region (such as strong glare or radar false alarm), the corresponding deviation term will increase, thereby driving the mode to adjust its representation in that region through backpropagation. Modes with larger reliability weights contribute more to the average value and loss, and can play a role in pulling back unreliable modes in weak evidence scenarios, reducing conflicts and high confidence errors among multiple modes.

[0069] Furthermore, before constructing the total loss function, it is necessary to obtain the reliability weights of each mode. The modal reliability assessment and gating module estimates the reliability weights of each mode based on the current environmental state, sensor self-diagnostic information, and multimodal feature statistics, and adjusts the adapter output and consistency loss accordingly by weighting or gating.

[0070] Specifically, obtaining the reliability weights of each mode includes the following steps: First, for each mode... Construct the corresponding reliability assessment input vector .

[0071] The reliability assessment input vector The information includes the following three categories: Scene status information, including day / night conditions, weather type, road slipperiness, and other environmental information; this information reflects the degree of impact of the current scene on each modal sensor. For example, the reliability of the camera modality may decrease in nighttime scenes, and the reliability of the LiDAR modality may decrease in rainy or foggy weather. Sensor self-diagnostic information, including occlusion alarms, fault codes, overheating, and other sensor status information; this information reflects the working status of the sensor itself. When the sensor malfunctions, the reliability of the output data will decrease. Modal characteristic statistics: for camera modalities, these include average image brightness, brightness variance, overexposed and underexposed pixel ratios, etc.; for LiDAR modalities, these include the number of effective points, point cloud density, proportion of distant points, and noise ratio, etc.; for 4D radar modalities, these include the number of effective targets, average signal-to-noise ratio, and false alarm density, etc.

[0072] Furthermore, the reliability assessment input vector can be represented as: Secondly, the reliability assessment input vector Inputting the reliability assessment network yields a reliability score for the corresponding mode. Specifically, this involves inputting the reliability assessment input vector... Input a lightweight evaluation network A non-negative reliability score was obtained. ,in, An activation function can be added to a 1-2 layer fully connected network, or it can be a rule-based linear function.

[0073] Furthermore, the reliability scores for all modes are normalized to obtain the reliability weights for each mode. : ;in, To prevent tiny constants with a denominator of zero.

[0074] It should be noted that, through normalization, the sum of the reliability weights of each modality is 1, which facilitates subsequent weighting operations in loss calculation and feature fusion.

[0075] Specifically, the application of the modal reliability weights includes: using the reliability weights to perform weighted or gated processing on the outputs of each modal adapter. During feature fusion, the reliability weights are utilized... Weighted averaging or gating is applied to the bird's-eye view features of different modalities; when the reliability weight of a certain modality... When the threshold value is below the preset threshold, the contribution of the mode is reduced or turned off, thereby automatically reducing the impact of unreliable modes in scenarios such as severe weather and sensor degradation.

[0076] In this embodiment, constructing the total loss function means weighting and summing the basic detection loss and various consistency losses to obtain the total loss function, as shown in the following formula: ;in, This is the total loss function; These are the weighting coefficients for each loss term.

[0077] It should be noted that, addressing the lack of a unified constraint mechanism combining geometric error, velocity estimation, and cross-modal consistency in existing transfer and fine-tuning methods, three types of physical consistency constraints are introduced simultaneously during the efficient parameter fine-tuning process: geometric consistency constraints use the geometrically relevant features or predictions of the frozen backbone output as a reference to limit the drift of 3D position and size predictions after fine-tuning; velocity consistency constraints use the velocity or Doppler information output by the 4D millimeter-wave radar as a reference to constrain the velocity prediction output by the detector head; and cross-modal consistency constraints constrain the occupancy results from the camera, LiDAR, and 4D millimeter-wave radar on a unified bird's-eye view grid. Since the above consistency constraints directly affect the key physical quantities related to 3D geometry and motion, even with reduced training epochs and greater gradient noise, they can effectively prevent the divergence of 3D position, size, and velocity predictions during the transfer process, thus maintaining the stability of 3D geometry and motion estimation while achieving efficient parameter fine-tuning.

[0078] S5: Update the adapter module and the low-rank parameterization module based on the total loss function to obtain the migration-fine-tuned 3D target detection model.

[0079] In this embodiment, the trainable parameters in the transfer subspace are updated using the total loss function to obtain the 3D target detection model after transfer fine-tuning. The gradient descent optimization algorithm is used to update only the adapter parameters in the representation-side transfer subspace and the low-rank parameters in the discriminant-side transfer subspace, while keeping the multimodal backbone network parameters and the detection head parameters of the unselected layers frozen.

[0080] Specifically, updating the adapter module and the low-rank parameterization module based on the total loss function includes the following steps: First, clarifying the range of frozen parameters and trainable parameters; specifically, the frozen parameters include: all parameters of the multimodal 3D target detection backbone network, including the parameters of the camera branch, LiDAR branch, 4D millimeter-wave radar branch, and unified bird's-eye view fusion module; and the parameters of the layers in the 3D target detection head that were not selected to apply low-rank parameterization.

[0081] The trainable parameters include: parameters characterizing each bird's-eye view adapter in the side-transfer subspace, including the weights and biases of dimensionality reduction units, nonlinear transformation units, and dimensionality increase units in modality-specific and shared adapters; and low-rank parameters discriminating each output branch in the side-transfer subspace, including the low-rank increment matrices in the 3D bounding box branch, orientation and velocity branch, and occupancy attribute branch. and .

[0082] Furthermore, all trainable parameters are collectively referred to as The parameters This constitutes the optimization object for the efficient migration and fine-tuning of parameters in this invention.

[0083] It should be noted that, due to the trainable parameters It only includes adapter parameters and low-rank matrix parameters, which are much fewer than the number of parameters in a complete network. Therefore, the computational and storage overhead during parameter update is reduced.

[0084] Furthermore, for trainable parameters Gradient descent updates are performed, and the specific process is as follows: Forward computation: Input a batch of training samples, and execute the processing steps S1 to S4 sequentially to obtain the total loss function. .

[0085] Backpropagation: After obtaining the total loss function value, the backpropagation algorithm is used to calculate the gradient of the total loss with respect to the trainable parameters. According to the chain rule, the gradient is calculated from the total loss function... Starting from the beginning, the algorithm backpropagates along the computation graph, calculating the gradients of each intermediate variable in turn, and finally obtaining the total loss with respect to the trainable parameters. gradient .

[0086] Parameter Update: After obtaining the gradient, the trainable parameters are updated using the gradient descent optimization algorithm, where the current iteration number is t and the learning rate is... The update rule for trainable parameters is then: ;in, These are the trainable parameters at the t-th iteration; These are the trainable parameters updated at the (t+1)th iteration; The learning rate controls the step size for each parameter update; Let be the gradient of the total loss with respect to the trainable parameters at the t-th iteration. In an alternative embodiment, in addition to stochastic gradient descent, adaptive learning rate optimization algorithms such as Adam can be used for parameter updates to achieve more stable convergence.

[0087] The gradient descent update process is repeated until the preset number of training rounds or validation set performance converges.

[0088] Furthermore, after each training round, the detection performance of the current model can be evaluated on the validation set, including metrics such as average accuracy, position error, size error, and velocity error. When the validation set performance no longer improves or reaches the preset training round, the training process is stopped. After training is completed, the updated adapter parameters and low-rank parameters are combined with the frozen multimodal backbone network parameters to obtain the transfer-tuned 3D object detection model.

[0089] The fine-tuned 3D target detection model includes: a parameter-frozen multimodal 3D target detection backbone network, including feature extraction subnetworks for each modality and a multimodal bird's-eye view feature fusion subnetwork; a trained representation-side transfer subspace, including modality-specific adapters and shared adapters; and a trained discriminative-side transfer subspace, including low-rank parameterization modules for each output branch.

[0090] After training is completed, the updated adapter parameters and low-rank parameters are deployed together with the frozen multimodal backbone network parameters to the vehicle computing platform. During actual operation, only forward inference is performed, and no further parameter updates are performed, which enables multimodal 3D target detection in different cities, different sensor configurations, and different weather scenarios.

[0091] Specifically, the forward inference process in the deployment phase includes: the vehicle-mounted multimodal sensors acquire camera images, LiDAR point clouds, and 4D radar data in real time during vehicle operation; the preprocessing and calibration module performs time synchronization, external parameter transformation, and basic filtering on the data of each modality, and unifies them into the vehicle or world coordinate system.

[0092] The camera branch extracts convolutional features from the image and maps the features to the bird's-eye view grid through geometric projection or viewpoint transformation modules; the LiDAR branch performs voxelization or sparse convolution on the point cloud to obtain LiDAR bird's-eye view features; the 4D radar branch encodes radar points or radar grids to generate radar bird's-eye view features; in each modal bird's-eye view path and the unified bird's-eye view fusion path, the pre-trained bird's-eye view adapter is used to perform dimensionality reduction, nonlinear transformation and dimensionality increase on the features, and the updated bird's-eye view features are generated through residual connections.

[0093] The features of each modality of bird's-eye view are fused into a unified bird's-eye view representation according to a preset method. The unified bird's-eye view representation is input into the 3D detection head after low-rank parameterization adjustment. The 3D bounding box branch outputs the category, 3D center and size of each candidate target, the orientation and velocity branch outputs the target orientation angle and velocity vector, and the occupancy attribute branch outputs the scene occupancy map and other semantic attributes.

[0094] Candidate detection results are filtered by confidence threshold and non-maximum suppression. The results are then converted from the vehicle coordinate system to the map coordinate system as needed. The output is a multimodal 3D target detection result containing information such as category, 3D position, size, orientation and speed.

[0095] It should be noted that, in response to the problem that existing multimodal 3D target detection networks require large-scale parameter fine-tuning and have high adaptation costs when adapting to different scenes and sensor configurations, this paper proposes to construct a representation-side transfer subspace and a discriminant-side transfer subspace. Under the premise of freezing the multimodal backbone network, only a small number of adapter parameters and low-rank parameters are updated. Since the number of newly added trainable parameters is less than the number of complete network parameters, and the training rounds can be shortened, adaptation can be completed by fine-tuning a small number of parameters in a short time when migrating between different cities and different sensor configurations. This reduces the computational resources and time costs of cross-scene and cross-sensor configuration adaptation, making it possible to achieve fast and stable cross-scene and cross-sensor configuration migration under vehicle computing power and limited annotation conditions.

[0096] On the other hand, this embodiment also provides an efficient transfer and fine-tuning system for three-dimensional target detection parameters for intelligent vehicles, which includes: a target detection backbone network module, which acquires and preprocesses multimodal sensor data of the vehicle to obtain multimodal data in a unified coordinate system, inputs the multimodal data into a multimodal three-dimensional target detection backbone network with frozen parameters and performs forward inference to obtain bird's-eye view features of each modality and unified bird's-eye view fusion features.

[0097] The representation-side transfer module inserts an adapter module into the paths of each modal bird's-eye view feature and the unified bird's-eye view fusion feature to fine-tune the each modal bird's-eye view feature and the unified bird's-eye view fusion feature, thereby constructing a representation-side transfer subspace.

[0098] The discriminative side migration module applies a low-rank parameterization module to a preset layer of the 3D target detection head, incrementally updates the weights of the 3D target detection head, and constructs a discriminative side migration subspace.

[0099] The constraint construction module calculates the basic detection loss and consistency constraint loss and constructs the total loss function based on the representation-side migration subspace and the discrimination-side migration subspace.

[0100] The migration fine-tuning module updates the adapter module and the low-rank parameterization module based on the total loss function to obtain the migration-fine-tuned 3D target detection model.

[0101] If the above functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0103] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0104] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0105] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for efficient transfer and fine-tuning of 3D target detection parameters for intelligent vehicles, characterized in that, include: Multimodal sensor data of the vehicle is acquired and preprocessed to obtain multimodal data in a unified coordinate system. The multimodal data is then input into a multimodal 3D target detection backbone network with frozen parameters and forward inference is performed to obtain bird's-eye view features of each modality and unified bird's-eye view fusion features. Adapter modules are inserted into the paths of each modal bird's-eye view feature and unified bird's-eye view fusion features to fine-tune the bird's-eye view features of each modality and the unified bird's-eye view fusion features, thereby constructing a representation-side transfer subspace. A low-rank parameterization module is applied to a preset layer of the 3D target detection head to incrementally update the weights of the 3D target detection head and construct a discriminative-side transfer subspace. Based on the representation-side transfer subspace and the discriminative-side transfer subspace, the basic detection loss and consistency constraint loss are calculated and a total loss function is constructed. Based on the total loss function, the adapter module and the low-rank parameterization module are updated to obtain the 3D target detection model after transfer fine-tuning.

2. The efficient transfer and fine-tuning method for 3D target detection parameters for intelligent vehicles as described in claim 1, characterized in that: Fine-tuning the modal bird's-eye view features and the unified bird's-eye view fusion features includes inserting modality-specific adapters into the paths of the modality-specific bird's-eye view features and inserting a shared adapter into the path of the unified bird's-eye view fusion features; using the dimensionality reduction units in the modality-specific adapters and the shared adapters to compress the number of channels of the input features to a preset number of channels; and applying nonlinear activation to the compressed features through a nonlinear transformation unit to obtain activated features. The number of channels of the activated feature is restored to the original number of channels by the upscaling unit to obtain the upscaling feature; the upscaling feature is then added element-wise to the input feature by the residual connection unit to obtain the fine-tuned bird's-eye view feature.

3. The efficient transfer and fine-tuning method for 3D target detection parameters for intelligent vehicles as described in claim 2, characterized in that: The construction of the discriminative side migration subspace includes determining a preset layer in the three-dimensional target detection head; The preset layer includes a 3D bounding box branch layer, an orientation and velocity branch layer, and an occupancy attribute branch layer; For the original weight matrix in the preset layer, generate a first low-rank matrix and a second low-rank matrix corresponding to the original weight matrix; Incremental weights are calculated based on the first low-rank matrix and the second low-rank matrix. The incremental weights are then combined with the frozen original weight matrix to obtain an updated weight matrix. Corresponding rank values ​​are configured for each preset layer. The rank values ​​configured for the 3D bounding box branch layer and the orientation and velocity branch layer are greater than the rank value of the occupancy attribute branch layer.

4. The efficient transfer and fine-tuning method for 3D target detection parameters for intelligent vehicles as described in claim 3, characterized in that: The consistency constraint loss includes geometric consistency loss, velocity consistency loss, and cross-modal consistency loss; calculating the consistency constraint loss includes obtaining the geometric reference prediction result output by the multimodal 3D target detection backbone network with frozen parameters, obtaining the current geometric prediction result output after being updated by the representation-side migration subspace and the discriminant-side migration subspace, and calculating the geometric consistency loss based on the geometric reference prediction result and the current geometric prediction result. The target velocity information in the multimodal sensor data is obtained as a velocity reference value, and the target velocity prediction value output by the three-dimensional target detection head is obtained. The velocity consistency loss is calculated based on the velocity reference value and the target velocity prediction value. The occupancy probability inferred by each modality is obtained in the unified bird's-eye view grid. The weighted average occupancy probability is calculated based on the reliability weight of each modality. The cross-modal consistency loss is calculated based on the occupancy probability of each modality and the weighted average occupancy probability.

5. The efficient transfer and fine-tuning method for 3D target detection parameters for intelligent vehicles as described in claim 4, characterized in that: Obtaining the reliability weights for each modality includes: for each modality, collecting environmental state information and sensor self-diagnostic information of the current scene, and extracting feature statistics for each modality to construct a reliability assessment input vector; wherein, the environmental state information includes weather conditions and illumination information; inputting the reliability assessment input vector into the reliability assessment network to obtain the reliability score for each modality; and normalizing the reliability scores of all modalities to obtain the reliability weights for each modality.

6. The efficient transfer and fine-tuning method for 3D target detection parameters for intelligent vehicles as described in claim 5, characterized in that: The reliability weight is used to adjust the output of the adapter module and the consistency constraint loss. This includes, for each mode, calculating the adapter output features by weighting the reliability weight of each mode with the output features of the mode-specific adapter corresponding to the mode; and fusing the adapter output features of each mode so that the mode with high reliability produces a greater feature contribution in the representation-side migration subspace. The contributions of each mode in the geometric consistency loss, velocity consistency loss, and cross-modal consistency loss are weighted according to the reliability weight, so that the modes with high reliability have a greater proportion in loss optimization; when the reliability weight of a mode is less than a preset gating threshold, the output features of the mode-specific adapter of the mode are adjusted using the attenuation coefficient to reduce the contribution ratio of the mode in feature fusion and loss calculation.

7. The efficient transfer and fine-tuning method for 3D target detection parameters for intelligent vehicles as described in claim 6, characterized in that: Inserting modality-specific adapters into the paths of each modality's bird's-eye view feature, and inserting shared adapters into the paths of the unified bird's-eye view fusion feature, includes analyzing the feature layers in the paths of each modality's bird's-eye view feature, selecting feature layers with spatial resolution less than a preset resolution threshold and channel number greater than a preset channel number threshold as the insertion positions for the modality-specific adapters; and analyzing the feature layers in the paths of the unified bird's-eye view fusion feature, selecting feature layers with spatial resolution less than a preset resolution threshold and channel number greater than a preset channel number threshold as the insertion positions for the shared adapters. Insert the modality-specific adapter and the shared adapter at the insertion location.

8. A high-efficiency transfer and fine-tuning system for three-dimensional target detection parameters for intelligent vehicles, employing the method described in any one of claims 1-7, characterized in that: The target detection backbone network module acquires and preprocesses multimodal sensor data of the vehicle to obtain multimodal data in a unified coordinate system. The multimodal data is then input into the multimodal 3D target detection backbone network with frozen parameters and forward inference is performed to obtain bird's-eye view features of each modality and unified bird's-eye view fusion features. The representation-side transfer module inserts an adapter module into the paths of each modal bird's-eye view feature and the unified bird's-eye view fusion feature to fine-tune the each modal bird's-eye view feature and the unified bird's-eye view fusion feature, and constructs a representation-side transfer subspace. The discriminative side migration module applies a low-rank parameterization module to a preset layer of the 3D target detection head, incrementally updates the weights of the 3D target detection head, and constructs a discriminative side migration subspace. The constraint construction module calculates the basic detection loss and consistency constraint loss and constructs the total loss function based on the representation-side migration subspace and the discrimination-side migration subspace. The migration fine-tuning module updates the adapter module and the low-rank parameterization module based on the total loss function to obtain the migration-fine-tuned 3D target detection model.

9. A computer device, comprising: A memory and a processor; the memory stores a computer program, characterized in that: when the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images