A V2X cooperative perception method and system based on lightweight domain adapter

Abstract

The present invention discloses a V2X collaborative perception method and system based on a lightweight domain adapter. The present invention places a lightweight domain adapter network in the feature map input branch of each collaborative vehicle of the first V2X collaborative perception network carried by the ego vehicle to form a second V2X collaborative perception network, and uses the trained second V2X collaborative perception network to realize collaborative perception of the ego vehicle and the collaborative vehicle. By introducing a domain adapter network, the present invention successfully realizes collaborative 3D target detection in the case of heterogeneous and heterogeneous feature maps of the ego vehicle and the collaborative vehicle. Without the need to retrain the entire basic collaborative detection network, the present invention effectively reduces the domain differences between heterogeneous and heterogeneous feature maps by inserting domain adapters corresponding to different feature maps, greatly improving the detection performance and scene adaptability of V2X-based collaborative 3D target detection, and has high practical value for improving the collaborative perception performance of autonomous driving.

Description

Technical Field

[0001] The present invention relates to a V2X collaborative perception method in the field of connected autonomous vehicle technology, and in particular to a V2X collaborative perception method and system based on a lightweight domain adapter. Background Art

[0002] V2X collaborative perception refers to a self-driving vehicle utilizing information received from nearby vehicles or road infrastructure for a more comprehensive observation. Using appropriate collaborative strategies, multiple views are integrated to compensate for information lost from a single perspective, enabling autonomous vehicles to more accurately understand their surroundings and thus enhance individual vehicle perception capabilities. Current V2X fusion strategies primarily include RGB camera image fusion and LiDAR point cloud fusion. Camera sensors can produce high-resolution RGB images but are sensitive to light and lack depth information. LiDAR sensors, on the other hand, can obtain precise depth information about objects but are sensitive to weather and lack semantically meaningful information such as color and texture. In practice, different automakers choose to deploy different types of sensors and perception methods based on their specific circumstances.

[0003] Existing V2X fusion methods rarely consider heterogeneous sensors and perception methods. By default, the sensors and feature encoders of both the cooperative vehicle and the ego vehicle are identical, significantly limiting the application and widespread adoption of V2X collaboration. In this context, finding a way to efficiently and cost-effectively fuse heterogeneous feature maps for environmental perception is crucial. Summary of the Invention

[0004] The purpose of the present invention is to overcome the problems of reduced target detection performance and poor detection results in existing V2X collaborative technologies due to heterogeneous sensors and heterogeneous feature encoders, and to provide a V2X collaborative perception method and system based on a lightweight domain adapter. By introducing a lightweight domain adapter, the present invention effectively reduces the domain differences between heterogeneous / heterogeneous feature maps without changing the existing basic collaborative detection network of the ego vehicle, thereby solving the problem of significant degradation in 3D target detection performance due to feature map heterogeneity, improving the scenario adaptability and generalization capabilities of V2X-based collaborative 3D target detection, and successfully realizing collaborative 3D target detection in the presence of heterogeneous feature maps.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] 1. A V2X cooperative perception method based on lightweight domain adapter

[0007] 1) The ego vehicle is equipped with a first V2X collaborative perception network, which includes a feature encoder for the ego vehicle, a V2X fusion backbone network, and a detection head. The feature encoders installed in multiple cooperative vehicles are the same as the feature encoder for the ego vehicle. Feature maps output by the feature encoders of all cooperative vehicles are sent to the ego vehicle and concatenated with the output of the feature encoder of the ego vehicle before being input into the V2X fusion backbone network. The V2X fusion backbone network is connected to the detection head.

[0008] 2) Freeze the network parameters of all feature encoders, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders, and train the V2X fusion backbone network and detection head until the training is complete. The trained V2X fusion backbone network and detection head are obtained and the first V2X cooperative perception network is updated.

[0009] 3) Adding a corresponding domain adapter network to the feature map input branch of each cooperative vehicle in the current first V2X cooperative perception network to form a second V2X cooperative perception network; for each cooperative vehicle, changing its sensor type and replacing its feature encoder simultaneously, or changing only the cooperative vehicle's feature encoder without changing its sensor type, so that the feature encoders of all cooperative vehicles are different from the feature encoder of the ego vehicle;

[0010] 4) Freeze the network parameters of all feature encoders, the V2X fusion backbone network, and the detection head. Combine the internal and external calibration parameters of the sensors between the cooperative vehicle and the ego vehicle, as well as the sensor data corresponding to the ego vehicle and all cooperative vehicles, and train each domain adapter network until training is complete. Obtain all trained domain adapter networks and update the second V2X cooperative perception network.

[0011] 5) Based on the feature maps collected and extracted by all cooperative vehicles in step 3) and the sensor data collected by the self-vehicle, the current second V2X cooperative perception network is used to perform vehicle cooperative perception to obtain a vehicle cooperative perception result.

[0012] The feature encoder of the ego vehicle and the feature encoder of the cooperative vehicle are pre-trained feature encoders carried by the ego vehicle / cooperative vehicle itself, or are obtained after the ego vehicle / cooperative vehicle trains its own feature encoder using its own sensor data.

[0013] The specific embodiment of 4) is:

[0014] 4.1) During each training session, the projection matrix and affine transformation matrix between each cooperative vehicle and the ego vehicle are calculated based on the intrinsic and extrinsic calibration parameters of the sensors between each cooperative vehicle and the ego vehicle;

[0015] 4.2) Using the collaborative vehicle and each ego vehicle’s collaborative ground truth, the single vehicle’s ground truth, and the affine transformation matrix, filter out the common areas and generate a mask corresponding to each ego vehicle;

[0016] 4.3) Freeze all feature encoders and the network parameters of the V2X fusion backbone network and detection head, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders respectively, and then use the projection matrix of each cooperative vehicle to project the feature map outputted by the corresponding feature encoder from the bird's-eye view to the ego vehicle coordinate system to obtain the corresponding projected feature map; then, perform the dot product of the projected feature map corresponding to each cooperative vehicle with the mask to obtain the corresponding feature map retaining only the common area, perform the dot product of the feature map outputted by the ego vehicle's feature encoder from the bird's-eye view with the mask to obtain the corresponding feature map retaining only the common area and use it as the true value of the common area of ​​the feature map; finally, calculate the mean square error loss of the common area of ​​the feature map of the ego vehicle and the cooperative vehicle based on the true value of the common area of ​​the feature map and the feature map retaining only the common area of ​​each cooperative vehicle and use it as the total loss to complete one training session;

[0017] 4.4) Repeat 4.1)-4.3) to train each domain adapter network until the training is completed, obtain the trained domain adapter network and update the second V2X collaborative perception network.

[0018] The specific embodiment of 4) is:

[0019] 4.1) During each training session, the projection matrix and affine transformation matrix between each cooperative vehicle and the ego vehicle are calculated based on the intrinsic and extrinsic calibration parameters of the sensors between each cooperative vehicle and the ego vehicle;

[0020] 4.2) Using the collaborative vehicle and each ego vehicle’s collaborative ground truth, the single vehicle’s ground truth, and the affine transformation matrix, filter out the common areas and generate a mask corresponding to each ego vehicle;

[0021] 4.3) Freeze all feature encoders and the network parameters of the V2X fusion backbone network and the detection head, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders respectively, and then use the projection matrix of each cooperative vehicle to project the feature map under the bird's-eye view output of the corresponding domain adapter network to the ego vehicle coordinate system to obtain the corresponding projected feature map; then, the corresponding feature map of only the common area is obtained by performing the dot product of the projected feature map corresponding to each cooperative vehicle with the mask, and the corresponding feature map of only the common area is obtained by performing the dot product of the feature map under the bird's-eye view output of the feature encoder of the ego vehicle with the mask and used as the true value of the common area of ​​the feature map. Finally, the mean square error loss of the common area of ​​the feature map of the ego vehicle and the cooperative vehicle is calculated based on the true value of the common area of ​​the feature map and the feature map of only the common area of ​​each cooperative vehicle, and the total loss is formed together with the classification loss, regression loss and direction loss of the detection head to complete one training;

[0022] 4.4) Repeat 4.1)-4.3) to train each domain adapter network until the training is completed, obtain the trained domain adapter network and update the second V2X collaborative perception network.

[0023] The domain adapter network includes a 3×3 convolution block, a 1×1 convolution block, a 1×1 convolution layer, a downsampling layer and an upsampling layer. The input of the domain adapter network is used as the input of the first 3×3 convolution block. The first 3×3 convolution block is sequentially connected to the first 1×1 convolution block, the downsampling layer, the second 3×3 convolution block, the second 1×1 convolution block, the upsampling layer and the third 3×3 convolution block, and then connected to the third 1×1 convolution block. The output of the first 1×1 convolution block is spliced ​​with the output of the third 1×1 convolution block and then input into the first 1×1 convolution layer. The output of the first 1×1 convolution layer is used as the output of the domain adapter network.

[0024] The 3×3 convolution block includes a 3×3 convolution layer, a batch normalization layer, and a RELU activation function layer connected in sequence; the 1×1 convolution block includes a 1×1 convolution layer, a batch normalization layer, and a RELU activation function layer connected in sequence.

[0025] 2. A V2X Collaborative Perception System Based on Lightweight Domain Adaptor

[0026] A first training unit is configured to train a V2X fusion backbone network and a detection head in the first V2X cooperative sensing network to obtain corresponding trained network parameters;

[0027] a second training unit, configured to construct a second V2X cooperative sensing network based on the trained network parameters corresponding to the V2X fusion backbone network and the detection head, and train a domain adapter network in the second V2X cooperative sensing network to obtain a trained second V2X cooperative sensing network;

[0028] The vehicle cooperative perception output unit is used to input the feature maps collected and extracted by all cooperative vehicles and the sensor data collected by the self-vehicle into the trained second V2X cooperative perception network, and output the vehicle cooperative perception results.

[0029] 3. A computer device

[0030] The device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the V2X collaborative perception method based on a lightweight domain adapter when executing the computer program.

[0031] 4. A Computer-Readable Storage Medium

[0032] The medium stores a computer program, which, when executed by a processor, implements the steps of the V2X collaborative perception method based on a lightweight domain adapter.

[0033] 5. A computer program product

[0034] The product includes a computer program / instruction, which, when executed by a processor, implements the steps of the V2X collaborative perception method based on a lightweight domain adapter.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. This invention implements 3D object detection based on V2X collaboration when the feature map types received by the ego vehicle by default are heterogeneous with those sent by the cooperative vehicle. This overcomes the limitation of existing V2X collaboration methods that only consider the processing of homogeneous feature maps but fail to cover the processing of heterogeneous feature maps, greatly improving scenario generalization.

[0037] 2. The present invention only uses convolutional networks in neural networks, and the number of network layers is small. It is a lightweight domain adapter. Compared with the domain adapter using the transformer structure, it has less computational overhead and memory usage, and has faster computing speed, more stable training, and easier model convergence.

[0038] 3. Once trained, the domain adapter designed in this invention can be plugged and played as a plug-in between the V2X fusion backbone network and the feature encoder. This eliminates the need to modify or retrain the underlying collaborative detection network structure. When new collaborative feature maps require integration, a new domain adapter needs only to be trained for the existing network. This approach is more practical and flexible than existing methods that require retraining.

[0039] 4. The present invention proposes the concept of common area and applies it to the loss function. The mean square error loss is only calculated for the common area of ​​the feature map between the self-vehicle and the cooperative vehicle, which more reasonably narrows the difference between different domains instead of forced alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention and better reflect the innovation, practicality and basic technical principles of the invention, the present invention is further described in detail below with reference to the accompanying drawings.

[0041] Figure 1 The figure is a flow chart of the overall method of the present invention.

[0042] Figure 2 This is the point cloud feature map of the self-vehicle in an embodiment of the present invention.

[0043] Figure 3 This is the collaborative vehicle point cloud feature map of an embodiment of the present invention.

[0044] Figure 4 This is the heterogeneous point cloud feature map of the collaborative vehicle in an embodiment of the present invention.

[0045] Figure 5 This is a diagram of the network structure of a lightweight domain adapter according to an embodiment of the present invention.

[0046] Figure 6 This is a mask map for generating a common area according to an embodiment of the present invention.

[0047] Figure 7 Schematic diagram showing the effects of having or not having a domain adapter and different training methods of the domain adapter according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0049] The embodiment and implementation process of the complete method according to the invention content of the present invention are as follows:

[0050] Example 1

[0051] The present invention proposes a V2X collaborative sensing method based on a lightweight domain adapter, comprising the following steps:

[0052] 1) The ego vehicle is equipped with a first V2X cooperative perception network, which includes a feature encoder for the ego vehicle, a V2X fusion backbone network, and a detection head. The feature encoders installed in multiple cooperative vehicles are the same as the feature encoder for the ego vehicle. The ego vehicle uses a lidar sensor, and PointPillars is used as the feature encoder for the radar point cloud. The feature map output by PointPillars from a bird's-eye view is as follows: Figure 2 As shown, the collaborative vehicle point cloud feature map is as follows Figure 3 As shown in Figure 2. The feature maps output by the feature encoders of all cooperating vehicles are sent to the ego vehicle, concatenated with the output of the ego vehicle's feature encoder, and then input into the V2X fusion backbone network, which is connected to the detection head. The ego vehicle's sensor type is LiDAR or RGB camera. The sensor data in steps 2) and 4) is data collected historically by the vehicle or extracted from a dataset.

[0053] The feature encoder of the ego vehicle and the feature encoder of the cooperative vehicle are pre-trained feature encoders carried by the ego vehicle / cooperative vehicle itself, or are obtained after the ego vehicle / cooperative vehicle trains its own feature encoder using its own sensor data.

[0054] 2) Freeze the network parameters of all feature encoders, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders, and train the V2X fusion backbone network and detection head until the training is complete. The trained V2X fusion backbone network and detection head are obtained and the first V2X cooperative perception network is updated.

[0055] 3) If Figure 1 As shown, a corresponding domain adapter network is added to the feature map input branch of each cooperative vehicle in the current first V2X cooperative perception network to form a second V2X cooperative perception network; for each cooperative vehicle, its sensor type is changed while the feature encoder is replaced, or only the feature encoder of the cooperative vehicle is changed without changing its sensor type, so that the feature encoders of all cooperative vehicles are different from the feature encoders of the ego vehicle; therefore, compared with the feature map of the ego vehicle from a bird's-eye view (i.e., the BEV feature map), the feature map extracted by the feature encoder of the cooperative vehicle from a bird's-eye view is heterogeneous or heterogeneous. In this embodiment, the cooperative vehicle adopts a lidar as a sensor and selects SECOND as a feature encoder. The feature map from a bird's-eye view output by SECOND is as follows: Figure 4 shown.

[0056] like Figure 5As shown in the figure, the domain adapter network includes a 3×3 convolutional block, a 1×1 convolutional block, a 1×1 convolutional layer, a downsampling layer, and an upsampling layer. The input of the domain adapter network serves as the input of the first 3×3 convolutional block. The first 3×3 convolutional block is sequentially connected to the first 1×1 convolutional block, the downsampling layer, the second 3×3 convolutional block, the second 1×1 convolutional block, the upsampling layer, and the third 3×3 convolutional block before being connected to the third 1×1 convolutional block. The output of the first 1×1 convolutional block is concatenated with the output of the third 1×1 convolutional block and then input into the first 1×1 convolutional layer. The output of the first 1×1 convolutional layer serves as the output of the domain adapter network. Each 3×3 convolutional block includes a 3×3 convolutional layer, a batch normalization layer, and a RELU activation function layer connected in sequence; each 1×1 convolutional block includes a 1×1 convolutional layer, a batch normalization layer, and a RELU activation function layer connected in sequence.

[0057] 4) Freeze the network parameters of all feature encoders, the V2X fusion backbone network, and the detection head. Combine the internal and external calibration parameters of the sensors between the cooperative vehicle and the ego vehicle, as well as the sensor data corresponding to the ego vehicle and all cooperative vehicles, and train each domain adapter network until training is complete. Obtain all trained domain adapter networks and update the second V2X cooperative perception network.

[0058] 4) Specifically:

[0059] 4.1) During each training session, the projection matrix and affine transformation matrix between each cooperative vehicle and the ego vehicle are calculated based on the intrinsic and extrinsic calibration parameters of the sensors between each cooperative vehicle and the ego vehicle;

[0060] 4.2) Using the cooperative vehicle and each ego vehicle's collaborative annotation ground truth, the single vehicle annotation ground truth, and the affine transformation matrix, filter out the common areas, i.e., the foreground and background that can be observed by both the cooperative vehicle and the ego vehicle. The rest are non-common areas. Set the common areas to 1 and the non-common areas to 0, and generate a mask corresponding to each ego vehicle.

[0061] 4.3) Freeze all feature encoders and the network parameters of the V2X fusion backbone network and the detection head, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders of the second V2X cooperative perception network respectively, and then use the projection matrix of each cooperative vehicle to project the feature map under the bird's-eye view output of the corresponding feature encoder to the ego vehicle coordinate system to obtain the corresponding projected feature map; then, the dot product of the projected feature map corresponding to each cooperative vehicle and the mask is obtained to obtain the corresponding feature map that only retains the common area. The mask is as follows Figure 6 As shown in (a), the feature map of the bird's-eye view output by the feature encoder of the self-vehicle is dot-producted with the mask to obtain the corresponding feature map that only retains the common area and serves as the true value of the common area of ​​the feature map, as shown in Figure 6As shown in (b); Finally, the mean square error loss of the common area of ​​the feature map of the self vehicle and the cooperative vehicle is calculated based on the true value of the common area of ​​the feature map and the feature map of each cooperative vehicle that only retains the common area and is used as the total loss to complete one training;

[0062] The formula for mean square error loss MSE is as follows:

[0063]

[0064] Where n is the number of samples, y i is the actual value of the i-th sample, that is, the value of the feature map of the ego vehicle at that location, is the predicted value of the i-th sample, that is, the value of the feature map of the cooperative vehicle at that location.

[0065] 4.4) Repeat 4.1)-4.3) to train each domain adapter network until the training is completed, obtain the trained domain adapter network and update the second V2X collaborative perception network.

[0066] 5) Based on the feature maps collected and extracted by all cooperative vehicles in step 3) and the sensor data collected by the self-vehicle, the current second V2X cooperative perception network is used to perform vehicle cooperative perception to obtain a vehicle cooperative perception result.

[0067] The present invention also proposes a V2X cooperative perception system based on a lightweight domain adapter, the system comprising:

[0068] A first training unit is configured to train a V2X fusion backbone network and a detection head in the first V2X cooperative sensing network to obtain corresponding trained network parameters;

[0069] a second training unit, configured to construct a second V2X cooperative sensing network based on the trained network parameters corresponding to the V2X fusion backbone network and the detection head, and train a domain adapter network in the second V2X cooperative sensing network to obtain a trained second V2X cooperative sensing network;

[0070] The vehicle collaborative perception output unit is used to input the sensor data collected by the self-vehicle and all collaborative vehicles into the trained second V2X collaborative perception network respectively, and output the vehicle collaborative perception results.

[0071] Example 2

[0072] The present invention proposes a V2X collaborative sensing method based on a lightweight domain adapter, comprising the following steps:

[0073] 1) A first V2X collaborative perception network is installed in the ego vehicle, and the first V2X collaborative perception network includes a feature encoder of the ego vehicle, a V2X fusion backbone network and a detection head. The feature encoders installed in multiple collaborative vehicles are the same as the feature encoder of the ego vehicle. The feature maps output by the feature encoders of all collaborative vehicles are sent to the ego vehicle and spliced ​​with the output of the feature encoder of the ego vehicle and then input into the V2X fusion backbone network. The V2X fusion backbone network is connected to the detection head; the sensor data in steps 2) and 4) are data collected historically by the vehicle or data extracted from the data set.

[0074] The feature encoder of the ego vehicle and the feature encoder of the cooperative vehicle are pre-trained feature encoders carried by the ego vehicle / cooperative vehicle itself, or are obtained after the ego vehicle / cooperative vehicle trains its own feature encoder using its own sensor data.

[0075] 2) Freeze the network parameters of all feature encoders, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders, and train the V2X fusion backbone network and detection head until the training is complete. The trained V2X fusion backbone network and detection head are obtained and the first V2X cooperative perception network is updated.

[0076] 3) If Figure 1 As shown, a corresponding domain adapter network is added to the feature map input branch of each cooperative vehicle in the current first V2X cooperative perception network to form a second V2X cooperative perception network; for each cooperative vehicle, its sensor type is changed while the feature encoder is replaced, or only the feature encoder of the cooperative vehicle is changed without changing its sensor type, so that the feature encoders of all cooperative vehicles are different from the feature encoders of the ego vehicle; therefore, compared with the feature map of the ego vehicle from a bird's-eye view (i.e., the BEV feature map), the feature map extracted by the feature encoder of the cooperative vehicle from a bird's-eye view is heterogeneous or heterogeneous. In this embodiment, the cooperative vehicle adopts a lidar as a sensor and selects SECOND as a feature encoder. The feature map from a bird's-eye view output by SECOND is as follows: Figure 4 shown.

[0077] like Figure 5As shown in the figure, the domain adapter network includes a 3×3 convolutional block, a 1×1 convolutional block, a 1×1 convolutional layer, a downsampling layer, and an upsampling layer. The input of the domain adapter network serves as the input of the first 3×3 convolutional block. The first 3×3 convolutional block is sequentially connected to the first 1×1 convolutional block, the downsampling layer, the second 3×3 convolutional block, the second 1×1 convolutional block, the upsampling layer, and the third 3×3 convolutional block before being connected to the third 1×1 convolutional block. The output of the first 1×1 convolutional block is concatenated with the output of the third 1×1 convolutional block and then input into the first 1×1 convolutional layer. The output of the first 1×1 convolutional layer serves as the output of the domain adapter network. Each 3×3 convolutional block includes a 3×3 convolutional layer, a batch normalization layer, and a RELU activation function layer connected in sequence; each 1×1 convolutional block includes a 1×1 convolutional layer, a batch normalization layer, and a RELU activation function layer connected in sequence.

[0078] 4) Freeze the network parameters of all feature encoders, the V2X fusion backbone network, and the detection head. Combine the internal and external calibration parameters of the sensors between the cooperative vehicle and the ego vehicle, as well as the sensor data corresponding to the ego vehicle and all cooperative vehicles, and train each domain adapter network until training is complete. Obtain all trained domain adapter networks and update the second V2X cooperative perception network.

[0079] 4) Specifically:

[0080] 4.1) During each training session, the projection matrix and affine transformation matrix between each cooperative vehicle and the ego vehicle are calculated based on the internal and external sensor calibration parameters between each cooperative vehicle and the ego vehicle;

[0081] 4.2) Using the cooperative vehicle and each ego vehicle's collaborative annotation ground truth, the single vehicle annotation ground truth, and the affine transformation matrix, filter out the common areas, i.e., the foreground and background that can be observed by both the cooperative vehicle and the ego vehicle. The rest are non-common areas. Set the common areas to 1 and the non-common areas to 0, and generate a mask corresponding to each ego vehicle.

[0082] 4.3) Freeze all feature encoders and the network parameters of the V2X fusion backbone network and the detection head, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders of the second V2X cooperative perception network respectively, and then use the projection matrix of each cooperative vehicle to project the feature map outputted by the corresponding feature encoder from the bird's-eye view to the ego vehicle coordinate system to obtain the corresponding projected feature map; then, the corresponding feature map of only the common area is obtained by performing the dot product of the projected feature map corresponding to each cooperative vehicle with the mask, and the corresponding feature map of only the common area is obtained by performing the dot product of the feature map outputted by the ego vehicle from the bird's-eye view with the mask and used as the true value of the common area of ​​the feature map; finally, the mean square error loss of the common area of ​​the feature map of the ego vehicle and the cooperative vehicle is calculated based on the true value of the common area of ​​the feature map and the feature map of only the common area of ​​each cooperative vehicle, and the total loss is formed together with the classification loss, regression loss and direction loss of the detection head, that is, the domain adapter is trained in a manner guided by the final detection result to complete one training session;

[0083] 4.4) Repeat 4.1)-4.3) to train each domain adapter network until the training is completed, obtain the trained domain adapter network and update the second V2X collaborative perception network.

[0084] 5) Based on the feature maps collected and extracted by all cooperative vehicles in step 3) and the sensor data collected by the self-vehicle, the current second V2X cooperative perception network is used to perform vehicle cooperative perception to obtain a vehicle cooperative perception result.

[0085] The present invention also proposes a V2X cooperative perception system based on a lightweight domain adapter, the system comprising:

[0086] A first training unit is configured to train a V2X fusion backbone network and a detection head in the first V2X cooperative sensing network to obtain corresponding trained network parameters;

[0087] a second training unit, configured to construct a second V2X cooperative sensing network based on the trained network parameters corresponding to the V2X fusion backbone network and the detection head, and train a domain adapter network in the second V2X cooperative sensing network to obtain a trained second V2X cooperative sensing network;

[0088] The vehicle cooperative perception output unit is used to input the feature maps collected and extracted by all cooperative vehicles and the sensor data collected by the self-vehicle into the trained second V2X cooperative perception network, and output the vehicle cooperative perception results.

[0089] Figure 7 The collaborative 3D target detection effect of the present invention on the OPV2V dataset is demonstrated. Figure 7(a) is the basic collaborative detection network of the ego vehicle, namely the PointPillars feature encoder, the V2X fusion backbone network Cobevt and the 1×1Conv detection head. The collaborative 3D target detection results after adding the heterogeneous feature map extracted by the SECOND feature encoder are detected without any processing. Figure 7 (b) shows the collaborative 3D object detection result after adding the heterogeneous feature map extracted by the SECOND feature encoder to the basic collaborative detection network with a non-end-to-end trained domain adapter. Figure 7 (c) shows the collaborative 3D object detection results after adding the heterogeneous feature maps extracted by the SECOND feature encoder to the basic collaborative detection network with an end-to-end trained domain adapter. Visualization shows that without the domain adapter, the heterogeneous feature maps result in four undetected objects in this scenario. With the non-end-to-end trained domain adapter, only two objects are undetected. With the end-to-end trained domain adapter, all objects are detected and the position parameters of the detection boxes are predicted more accurately.

[0090] Tables 1 and 2 list the quantitative results of 3D object detection based on a wider range of V2X collaborative fusion backbone networks using the OPV2V dataset. Three existing V2X collaborative fusion backbone networks were selected and subjected to collaborative detection inference with heterogeneous feature maps added to the input. These networks were tested without a domain adapter, with a non-end-to-end trained domain adapter, and with an end-to-end trained domain adapter. Table 1 shows that inserting a domain adapter before these V2X collaborative fusion backbone networks significantly improved the average precision (AP) of collaborative 3D object detection, regardless of whether it was trained end-to-end or end-to-end. The end-to-end trained domain adapter achieved the greatest improvement in AP.

[0091] Table 1 shows the collaborative 3D object detection results of the method in Example 1 of the present invention and the existing method without domain adapter on the OPV2V dataset.

[0092]

[0093] Table 2 shows the collaborative 3D object detection results of the method in Example 2 of the present invention and the existing method without domain adapter on the OPV2V dataset.

[0094]

[0095] To further demonstrate the effectiveness of the present invention, Table 3 lists the collaborative 3D object detection results after inserting an end-to-end trained domain adapter in front of the V2X fusion backbone network on another OPV2V test dataset (the number of cooperative vehicles is at least 3), with different heterogeneous feature map combinations of PointPillars, SECOND (①), PointPillars, Efficienetb1 (②), and PointPillars, SECOND, Efficienetb1 (③) as input.

[0096] Table 3 shows the collaborative 3D object detection results of the method in Example 2 of the present invention and the existing method without domain adapter on the OPV2V dataset.

[0097]

[0098] Finally, it should be noted that the above embodiments and explanations are intended only to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced with equivalents without departing from the spirit and scope of the technical solutions disclosed in the present invention, and all such modifications or equivalents should be included in the scope of protection of the claims of the present invention.

Claims

1. A V2X collaborative sensing method based on a lightweight domain adapter, characterized in that: The steps include: 1) The ego vehicle is equipped with a first V2X collaborative perception network, which includes a feature encoder for the ego vehicle, a V2X fusion backbone network, and a detection head. The feature encoders installed in multiple cooperative vehicles are the same as the feature encoder for the ego vehicle. The feature maps output by the feature encoders of all cooperative vehicles are sent to the ego vehicle, concatenated with the output of the feature encoder of the ego vehicle, and then input into the V2X fusion backbone network. The V2X fusion backbone network is connected to the detection head. 2) Freeze the network parameters of all feature encoders, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders, and train the V2X fusion backbone network and detection head until the training is complete. The trained V2X fusion backbone network and detection head are obtained and the first V2X cooperative perception network is updated; 3) Adding a corresponding domain adapter network to the feature map input branch of each cooperating vehicle in the current first V2X cooperative perception network to form a second V2X cooperative perception network; for each cooperating vehicle, changing its sensor type and replacing its feature encoder simultaneously, or changing only the cooperating vehicle's feature encoder without changing its sensor type, so that the feature encoders of all cooperating vehicles are different from the feature encoder of the ego vehicle; 4) Freeze the network parameters of all feature encoders, the V2X fusion backbone network, and the detection head. Combine the intrinsic and extrinsic calibration parameters of the sensors between the cooperative vehicle and the ego vehicle, as well as the sensor data corresponding to the ego vehicle and all cooperative vehicles, and train each domain adapter network until training is complete. Obtain all trained domain adapter networks and update the second V2X cooperative perception network. The specific details of 4) are: 4.1) During each training session, the projection matrix and affine transformation matrix between each cooperative vehicle and the ego vehicle are calculated based on the intrinsic and extrinsic calibration parameters of the sensors between each cooperative vehicle and the ego vehicle. 4.2) Using the collaborative vehicle and each ego vehicle’s collaborative ground truth, the single vehicle’s ground truth, and the affine transformation matrix, filter out the common areas and generate a mask corresponding to each ego vehicle; 4.3) Freeze the network parameters of all feature encoders, the V2X fusion backbone network, and the detection head. Input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders. Then, use the projection matrix of each cooperative vehicle to project the bird's-eye view feature map output by the corresponding feature encoder into the ego vehicle coordinate system to obtain the corresponding projected feature map. Then, the projected feature map corresponding to each cooperative vehicle is dot-producted with the mask to obtain the corresponding feature map that only retains the common area. The feature map output by the feature encoder of the ego vehicle from the bird's-eye view is dot-producted with the mask to obtain the corresponding feature map that only retains the common area and use it as the true value of the common area of ​​the feature map. Finally, based on the true value of the common area of ​​the feature map and the feature map that only retains the common area of ​​each cooperative vehicle, the mean squared error loss of the common area of ​​the feature map of the ego vehicle and the cooperative vehicle is calculated and used as the total loss to complete one training session. 4.4) Repeat 4.1)-4.3) to train each domain adapter network until training is complete, obtain the trained domain adapter network, and update the second V2X collaborative perception network; 5) Based on the feature maps collected and extracted by all cooperative vehicles in step 3) and the sensor data collected by the self-vehicle, the current second V2X cooperative perception network is used to perform vehicle cooperative perception to obtain a vehicle cooperative perception result.

2. The V2X collaborative sensing method based on a lightweight domain adapter according to claim 1, characterized in that: The feature encoder of the ego vehicle and the feature encoder of the cooperative vehicle are pre-trained feature encoders carried by the ego vehicle / cooperative vehicle itself, or are obtained after the ego vehicle / cooperative vehicle trains its own feature encoder using its own sensor data.

3. The V2X collaborative sensing method based on a lightweight domain adapter according to claim 1, characterized in that: The total loss also includes the classification loss, regression loss and direction loss of the detection head.

4. The V2X collaborative sensing method based on a lightweight domain adapter according to claim 1, characterized in that: The domain adapter network includes a 3×3 convolution block, a 1×1 convolution block, a 1×1 convolution layer, a downsampling layer and an upsampling layer. The input of the domain adapter network is used as the input of the first 3×3 convolution block. The first 3×3 convolution block is sequentially connected to the first 1×1 convolution block, the downsampling layer, the second 3×3 convolution block, the second 1×1 convolution block, the upsampling layer and the third 3×3 convolution block, and then connected to the third 1×1 convolution block. The output of the first 1×1 convolution block is spliced ​​with the output of the third 1×1 convolution block and then input into the first 1×1 convolution layer. The output of the first 1×1 convolution layer is used as the output of the domain adapter network.

5. The V2X collaborative sensing method based on a lightweight domain adapter according to claim 4, characterized in that: The 3×3 convolution block includes a 3×3 convolution layer, a batch normalization layer, and a RELU activation function layer connected in sequence; the 1×1 convolution block includes a 1×1 convolution layer, a batch normalization layer, and a RELU activation function layer connected in sequence.

6. A system for implementing V2X collaborative sensing based on a lightweight domain adapter according to any one of claims 1 to 5, characterized in that: include: A first training unit is configured to train a V2X fusion backbone network and a detection head in the first V2X cooperative sensing network to obtain corresponding trained network parameters; a second training unit, configured to construct a second V2X cooperative sensing network based on the trained network parameters corresponding to the V2X fusion backbone network and the detection head, and train a domain adapter network in the second V2X cooperative sensing network to obtain a trained second V2X cooperative sensing network; The vehicle collaborative perception output unit is used to input the sensor data collected by the self-vehicle and all collaborative vehicles into the trained second V2X collaborative perception network respectively, and output the vehicle collaborative perception results.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the V2X collaborative perception method based on a lightweight domain adapter according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the V2X collaborative perception method based on a lightweight domain adapter according to any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the V2X collaborative perception method based on a lightweight domain adapter as described in any one of claims 1 to 5 are implemented.

Images