V2X cooperative sensing method and system based on lightweight domain adapters

Abstract

The invention discloses a V2X cooperative sensing method and system based on a lightweight domain adapter. According to the invention, a lightweight domain adapter network is placed in a feature map input branch of each cooperative vehicle of a first V2X cooperative sensing network carried by an own vehicle to form a second V2X cooperative sensing network, and cooperative sensing of the own vehicle and the cooperative vehicles is realized by using the trained second V2X cooperative sensing network. According to the invention, by introducing the domain adapter network, collaborative 3D target detection under the condition of heterogeneous feature maps of the own vehicle and the collaborative vehicle is successfully realized. According to the method, under the condition that the whole basic cooperative detection network does not need to be retrained, the domain adapters corresponding to different feature maps are inserted, so that the domain difference between heterogeneous feature maps is effectively reduced, the detection performance and scene adaptability of cooperative 3D target detection based on V2X are greatly improved, and the detection efficiency is improved. The method has a high practical value for improving the collaborative perception performance of automatic driving.

Description

Technical Field

[0001] The present invention relates to a V2X collaborative perception method in the technical field of networked autonomous driving vehicles, and in particular to a V2X collaborative perception method and system based on a lightweight domain adapter. Background Art

[0002] V2X collaborative perception means that the self-vehicle uses the information received from nearby vehicles or road facilities to conduct more comprehensive observations, and fuses multiple views through appropriate collaborative strategies to compensate for the information loss caused by a single perspective, so that the autonomous driving vehicle can understand the surrounding environment more accurately, thereby improving the perception ability of a single vehicle. The current V2X fusion strategy mainly includes fusion based on RGB camera images and LiDAR point cloud. Camera sensors can obtain high-resolution RGB images, but are sensitive to light and lack depth information; while LiDAR sensors can obtain accurate depth information of objects, but are sensitive to weather and lack information such as color and texture that can represent semantics. In actual applications, different car manufacturers choose to configure different types of sensors and different perception methods according to their own conditions.

[0003] However, existing V2X fusion methods rarely consider the situation of heterogeneous sensors and heterogeneous perception methods. The sensors and feature encoders of cooperative vehicles and ego vehicles are of the same type by default, which greatly limits the application and promotion of V2X cooperative technology. In this context, how to reasonably fuse heterogeneous and heterogeneous feature maps for environmental perception at a very low cost is of high value. Summary of the invention

[0004] The purpose of the present invention is to overcome the problem of reduced target detection performance and poor detection effect of existing V2X collaborative technology due to heterogeneous sensors and heterogeneous feature encoders, and to provide a V2X collaborative perception method and system based on a lightweight domain adapter. By introducing a lightweight domain adapter, the present invention effectively reduces the domain difference between heterogeneous / heterogeneous feature maps without changing the existing basic collaborative detection network of the ego vehicle, thereby solving the problem of significant degradation of 3D target detection performance due to the heterogeneity of feature maps, improving the scene adaptability and generalization ability of V2X-based collaborative 3D target detection, and successfully realizing collaborative 3D target detection in the case of heterogeneous and heterogeneous feature maps.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] 1. A V2X cooperative perception method based on lightweight domain adapter

[0007] 1) The first V2X cooperative perception network is installed in the ego vehicle, and the first V2X cooperative perception network includes a feature encoder of the ego vehicle, a V2X fusion backbone network and a detection head. The feature encoders installed in multiple cooperative vehicles are the same as the feature encoder of the ego vehicle. The feature maps output by the feature encoders of all cooperative vehicles are sent to the ego vehicle and spliced ​​with the output of the feature encoder of the ego vehicle and then input into the V2X fusion backbone network. The V2X fusion backbone network is connected to the detection head;

[0008] 2) Freeze the network parameters of all feature encoders, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders, and train the V2X fusion backbone network and the detection head until the training is completed, obtain the trained V2X fusion backbone network and the detection head, and update the first V2X cooperative perception network;

[0009] 3) Adding a corresponding domain adapter network to the feature map input branch of each cooperative vehicle of the current first V2X cooperative perception network to form a second V2X cooperative perception network; for each cooperative vehicle, changing its sensor type and replacing the feature encoder at the same time, or changing only the feature encoder of the cooperative vehicle without changing its sensor type, so that the feature encoders of all cooperative vehicles are different from the feature encoder of the self vehicle;

[0010] 4) Freeze all feature encoders and the network parameters of the V2X fusion backbone network and the detection head, combine the sensor internal and external calibration parameters between the cooperative vehicle and the self-vehicle, and the sensor data corresponding to the self-vehicle and all cooperative vehicles, train each domain adapter network until the training is completed, obtain all trained domain adapter networks and update the second V2X cooperative perception network;

[0011] 5) Based on the feature maps collected and extracted by all cooperative vehicles in step 3) and the sensor data collected by the self-vehicle, the current second V2X cooperative perception network is used to perform vehicle cooperative perception to obtain vehicle cooperative perception results.

[0012] The feature encoder of the ego vehicle and the feature encoder of the cooperative vehicle are pre-trained feature encoders carried by the ego vehicle / cooperative vehicle itself, or are obtained after the ego vehicle / cooperative vehicle trains its own feature encoder using its own sensor data.

[0013] The specific aspects of 4) are:

[0014] 4.1) In each training, the projection matrix and affine transformation matrix between each cooperative vehicle and the self-vehicle are calculated based on the calibration parameters of the sensor internal and external parameters between each cooperative vehicle and the self-vehicle;

[0015] 4.2) Use the collaborative annotation truth values ​​of the collaborative vehicle and each ego vehicle, the single vehicle annotation truth value, and the affine transformation matrix to filter out the common area and generate a mask corresponding to each ego vehicle;

[0016] 4.3) Freeze all feature encoders and the network parameters of the V2X fusion backbone network and the detection head, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders respectively, and then use the projection matrix of each cooperative vehicle to project the feature map under the bird's-eye view output of the corresponding feature encoder to the ego vehicle coordinate system to obtain the corresponding projected feature map; then, dot product the projected feature map corresponding to each cooperative vehicle with the mask to obtain the corresponding feature map that only retains the common area, dot product the feature map under the bird's-eye view output of the feature encoder of the ego vehicle with the mask to obtain the corresponding feature map that only retains the common area and use it as the true value of the common area of ​​the feature map; finally, calculate the mean square error loss of the common area of ​​the feature map of the ego vehicle and the cooperative vehicle based on the true value of the common area of ​​the feature map and the feature map of only the common area corresponding to each cooperative vehicle and use it as the total loss to complete one training;

[0017] 4.4) Repeat 4.1)-4.3) to train each domain adapter network until the training is completed, obtain the trained domain adapter network and update the second V2X collaborative perception network.

[0018] The specific aspects of 4) are:

[0019] 4.1) In each training, the projection matrix and affine transformation matrix between each cooperative vehicle and the self-vehicle are calculated based on the calibration parameters of the sensor internal and external parameters between each cooperative vehicle and the self-vehicle;

[0020] 4.2) Use the collaborative annotation truth values ​​of the collaborative vehicle and each ego vehicle, the single vehicle annotation truth value, and the affine transformation matrix to filter out the common area and generate a mask corresponding to each ego vehicle;

[0021] 4.3) Freeze all feature encoders and the network parameters of the V2X fusion backbone network and the detection head, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders respectively, and then use the projection matrix of each cooperative vehicle to project the feature map under the bird's-eye view output of the corresponding domain adapter network to the ego vehicle coordinate system to obtain the corresponding projected feature map; then, the corresponding feature map of only the common area is obtained by performing the dot product of the projected feature map corresponding to each cooperative vehicle with the mask, and the corresponding feature map of only the common area is obtained by performing the dot product of the feature map under the bird's-eye view output of the feature encoder of the ego vehicle with the mask and used as the true value of the common area of ​​the feature map; finally, the mean square error loss of the common area of ​​the feature map of the ego vehicle and the cooperative vehicle is calculated based on the true value of the common area of ​​the feature map and the feature map of only the common area corresponding to each cooperative vehicle, and the total loss is formed together with the classification loss, regression loss and direction loss of the detection head to complete one training;

[0022] 4.4) Repeat 4.1)-4.3) to train each domain adapter network until the training is completed, obtain the trained domain adapter network and update the second V2X collaborative perception network.

[0023] The domain adapter network includes a 3×3 convolution block, a 1×1 convolution block, a 1×1 convolution layer, a downsampling layer and an upsampling layer. The input of the domain adapter network is used as the input of the first 3×3 convolution block. The first 3×3 convolution block is sequentially connected to the first 1×1 convolution block, the downsampling layer, the second 3×3 convolution block, the second 1×1 convolution block, the upsampling layer and the third 3×3 convolution block, and then connected to the third 1×1 convolution block. The output of the first 1×1 convolution block is spliced ​​with the output of the third 1×1 convolution block and then input into the first 1×1 convolution layer. The output of the first 1×1 convolution layer is used as the output of the domain adapter network.

[0024] The 3×3 convolution block includes a 3×3 convolution layer, a batch normalization layer, and a RELU activation function layer connected in sequence; the 1×1 convolution block includes a 1×1 convolution layer, a batch normalization layer, and a RELU activation function layer connected in sequence.

[0025] 2. A V2X cooperative perception system based on lightweight domain adapter

[0026] A first training unit is used to train a V2X fusion backbone network and a detection head in a first V2X collaborative perception network to obtain corresponding trained network parameters;

[0027] A second training unit is used to construct a second V2X collaborative perception network based on the trained network parameters corresponding to the V2X fusion backbone network and the detection head, and train the domain adapter network in the second V2X collaborative perception network to obtain a trained second V2X collaborative perception network;

[0028] The vehicle cooperative perception output unit is used to input the feature maps collected and extracted by all cooperative vehicles and the sensor data collected by the self-vehicle into the trained second V2X cooperative perception network, and output the vehicle cooperative perception results.

[0029] 3. A computer device

[0030] The device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the V2X collaborative perception method based on a lightweight domain adapter when executing the computer program.

[0031] 4. A computer-readable storage medium

[0032] The medium stores a computer program, and when the computer program is executed by the processor, the steps of the V2X collaborative perception method based on the lightweight domain adapter are implemented.

[0033] V. A computer program product

[0034] The product includes a computer program / instruction, which, when executed by a processor, implements the steps of the V2X collaborative perception method based on a lightweight domain adapter.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The present invention realizes 3D target detection based on V2X collaboration when the feature map type received by the self-vehicle by default and the feature map type sent by the cooperative vehicle are heterogeneous. This solves the limitation of the existing V2X collaboration method that only considers the processing of homogeneous feature maps but does not cover the processing of heterogeneous feature maps, and greatly improves the scene generalization.

[0037] 2. The present invention only uses convolutional networks in neural networks, and the number of network layers is small. It is a lightweight domain adapter. Compared with the domain adapter with transformer structure, it has less computing overhead and video memory occupancy resources, and has faster computing speed, more stable training, and easier model convergence.

[0038] 3. After training, the domain adapter designed by the present invention can be "plug-and-play" as a plug-in placed between the V2X fusion backbone network and the feature encoder, without changing the basic collaborative detection network structure of the vehicle itself or retraining the network. When a new collaborative feature map needs to be fused, it only needs to train a new domain adapter for the existing network. Compared with the existing method that requires retraining, it is more practical and more flexible.

[0039] 4. The present invention proposes the concept of common area and applies it to the loss function. The mean square error loss is calculated only for the common area of ​​the feature map between the self-vehicle and the cooperative vehicle, which more reasonably reduces the difference between different domains instead of forced alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solution of the embodiment of the present invention and better reflect the innovation and practicality of the invention as well as the basic technical principles, the present invention is further described in detail below with reference to the accompanying drawings.

[0041] Figure 1 The figure is a flow chart of the overall method of the present invention.

[0042] Figure 2 This is a point cloud feature map of the self-vehicle according to an embodiment of the present invention.

[0043] Figure 3 This is a collaborative vehicle point cloud feature map according to an embodiment of the present invention.

[0044] Figure 4 This is a heterogeneous point cloud feature map of collaborative vehicles in an embodiment of the present invention.

[0045] Figure 5 This is a diagram of the network structure of a lightweight domain adapter according to an embodiment of the present invention.

[0046] Figure 6 A mask map of a common area is generated according to an embodiment of the present invention.

[0047] Figure 7 1 is a diagram showing the effects of having or not having a domain adapter and different training methods of the domain adapter according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] The embodiment and implementation process of the complete method according to the invention content of the present invention are as follows:

[0050] Example 1

[0051] The present invention proposes a V2X collaborative perception method based on a lightweight domain adapter, comprising the following steps:

[0052] 1) The first V2X cooperative perception network is installed in the ego vehicle, and the first V2X cooperative perception network includes a feature encoder of the ego vehicle, a V2X fusion backbone network and a detection head. The feature encoders installed in multiple cooperative vehicles are the same as the feature encoder of the ego vehicle. The ego vehicle uses a lidar sensor, and PointPillars is used as a feature encoder of the radar point cloud. The feature map of the bird's-eye view output by PointPillars is as follows: Figure 2 As shown, the collaborative vehicle point cloud feature map is as follows Figure 3 As shown. The feature maps output by the feature encoders of all cooperative vehicles are sent to the ego vehicle and concatenated with the output of the feature encoder of the ego vehicle before being input into the V2X fusion backbone network, which is connected to the detection head; the sensor type of the ego vehicle is a lidar or an RGB camera. The sensor data in steps 2) and 4) are data collected historically by the vehicle or extracted from the data set.

[0053] The feature encoder of the ego vehicle and the feature encoder of the cooperative vehicle are pre-trained feature encoders carried by the ego vehicle / cooperative vehicle itself, or are obtained after the ego vehicle / cooperative vehicle trains its own feature encoder using its own sensor data.

[0054] 2) Freeze the network parameters of all feature encoders, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders, and train the V2X fusion backbone network and the detection head until the training is completed, obtain the trained V2X fusion backbone network and the detection head, and update the first V2X cooperative perception network;

[0055] 3) If Figure 1 As shown, a corresponding domain adapter network is added to the feature map input branch of each cooperative vehicle of the current first V2X cooperative perception network to form a second V2X cooperative perception network; for each cooperative vehicle, its sensor type is changed and the feature encoder is replaced at the same time, or only the feature encoder of the cooperative vehicle is changed without changing its sensor type, so that the feature encoders of all cooperative vehicles are different from the feature encoders of the ego vehicle; therefore, compared with the feature map of the ego vehicle from a bird's-eye view (i.e., the BEV feature map), the feature map extracted by the feature encoder of the cooperative vehicle from a bird's-eye view is heterogeneous or heterogeneous. In this embodiment, the cooperative vehicle adopts a lidar as a sensor, and selects SECOND as a feature encoder. The feature map from a bird's-eye view output by SECOND is as follows: Figure 4 shown.

[0056] like Figure 5As shown in the figure, the domain adapter network includes a 3×3 convolution block, a 1×1 convolution block, a 1×1 convolution layer, a downsampling layer and an upsampling layer. The input of the domain adapter network is used as the input of the first 3×3 convolution block. The first 3×3 convolution block is sequentially connected to the first 1×1 convolution block, the downsampling layer, the second 3×3 convolution block, the second 1×1 convolution block, the upsampling layer and the third 3×3 convolution block, and then connected to the third 1×1 convolution block. The output of the first 1×1 convolution block is concatenated with the output of the third 1×1 convolution block and then input to the first 1×1 convolution layer. The output of the first 1×1 convolution layer is used as the output of the domain adapter network. Each 3×3 convolution block includes a 3×3 convolution layer, a batch normalization layer and a RELU activation function layer connected in sequence; each 1×1 convolution block includes a 1×1 convolution layer, a batch normalization layer and a RELU activation function layer connected in sequence.

[0057] 4) Freeze all feature encoders and the network parameters of the V2X fusion backbone network and the detection head, combine the sensor internal and external calibration parameters between the cooperative vehicle and the self-vehicle, and the sensor data corresponding to the self-vehicle and all cooperative vehicles, train each domain adapter network until the training is completed, obtain all trained domain adapter networks and update the second V2X cooperative perception network;

[0058] 4) Specifically:

[0059] 4.1) In each training, the projection matrix and affine transformation matrix between each cooperative vehicle and the self-vehicle are calculated based on the calibration parameters of the sensor internal and external parameters between each cooperative vehicle and the self-vehicle;

[0060] 4.2) Use the collaborative annotation truth value of the collaborative vehicle and each ego vehicle, the single vehicle annotation truth value, and the affine transformation matrix to filter out the common area, that is, the foreground and background that can be observed by both the collaborative vehicle and the ego vehicle, and the rest are non-public areas. Set the public area to 1 and the non-public area to 0, and generate a mask corresponding to each ego vehicle;

[0061] 4.3) Freeze all feature encoders and the network parameters of the V2X fusion backbone network and the detection head, input the sensor data corresponding to the self-vehicle and all cooperative vehicles into the corresponding feature encoders of the second V2X cooperative perception network respectively, and then use the projection matrix of each cooperative vehicle to project the feature map under the bird's-eye view output of the corresponding feature encoder to the self-vehicle coordinate system to obtain the corresponding projected feature map; then, dot product the projected feature map corresponding to each cooperative vehicle with the mask to obtain the corresponding feature map that only retains the common area, and the mask is as follows Figure 6 As shown in (a); the feature map under the bird's-eye view output by the feature encoder of the ego vehicle is dot-producted with the mask to obtain the corresponding feature map that only retains the common area and serves as the true value of the common area of ​​the feature map, as shown in Figure 6As shown in (b); finally, the mean square error loss of the common area of ​​the feature map of the self vehicle and the cooperative vehicle is calculated based on the true value of the common area of ​​the feature map and the feature map of each cooperative vehicle that only retains the common area and is used as the total loss to complete a training;

[0062] The formula for mean square error loss MSE is as follows:

[0063]

[0064] Where n is the number of samples, y i is the actual value of the i-th sample, that is, the value of the feature map of the ego vehicle at that location, is the predicted value of the i-th sample, that is, the value of the feature map of the cooperative vehicle at that location.

[0065] 4.4) Repeat 4.1)-4.3) to train each domain adapter network until the training is completed, obtain the trained domain adapter network and update the second V2X collaborative perception network.

[0066] 5) Based on the feature maps collected and extracted by all cooperative vehicles in step 3) and the sensor data collected by the self-vehicle, the current second V2X cooperative perception network is used to perform vehicle cooperative perception to obtain vehicle cooperative perception results.

[0067] The present invention also proposes a V2X cooperative perception system based on a lightweight domain adapter, the system comprising:

[0068] A first training unit is used to train a V2X fusion backbone network and a detection head in a first V2X collaborative perception network to obtain corresponding trained network parameters;

[0069] A second training unit is used to construct a second V2X collaborative perception network based on the trained network parameters corresponding to the V2X fusion backbone network and the detection head, and train the domain adapter network in the second V2X collaborative perception network to obtain a trained second V2X collaborative perception network;

[0070] The vehicle collaborative perception output unit is used to input the sensor data collected by the self-vehicle and all collaborative vehicles into the trained second V2X collaborative perception network respectively, and output the vehicle collaborative perception results.

[0071] Example 2

[0072] The present invention proposes a V2X collaborative perception method based on a lightweight domain adapter, comprising the following steps:

[0073] 1) A first V2X collaborative perception network is installed in the ego vehicle, and the first V2X collaborative perception network includes a feature encoder of the ego vehicle, a V2X fusion backbone network and a detection head. The feature encoders installed in multiple collaborative vehicles are the same as the feature encoder of the ego vehicle. The feature maps output by the feature encoders of all collaborative vehicles are sent to the ego vehicle and spliced ​​with the output of the feature encoder of the ego vehicle and then input into the V2X fusion backbone network. The V2X fusion backbone network is connected to the detection head; the sensor data in steps 2) and 4) are data collected historically by the vehicle or data extracted from a data set.

[0074] The feature encoder of the ego vehicle and the feature encoder of the cooperative vehicle are pre-trained feature encoders carried by the ego vehicle / cooperative vehicle itself, or are obtained after the ego vehicle / cooperative vehicle trains its own feature encoder using its own sensor data.

[0075] 2) Freeze the network parameters of all feature encoders, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders, and train the V2X fusion backbone network and the detection head until the training is completed, obtain the trained V2X fusion backbone network and the detection head, and update the first V2X cooperative perception network;

[0076] 3) If Figure 1 As shown, a corresponding domain adapter network is added to the feature map input branch of each cooperative vehicle of the current first V2X cooperative perception network to form a second V2X cooperative perception network; for each cooperative vehicle, its sensor type is changed and the feature encoder is replaced at the same time, or only the feature encoder of the cooperative vehicle is changed without changing its sensor type, so that the feature encoders of all cooperative vehicles are different from the feature encoders of the ego vehicle; therefore, compared with the feature map of the ego vehicle from a bird's-eye view (i.e., the BEV feature map), the feature map extracted by the feature encoder of the cooperative vehicle from a bird's-eye view is heterogeneous or heterogeneous. In this embodiment, the cooperative vehicle adopts a lidar as a sensor, and selects SECOND as a feature encoder. The feature map from a bird's-eye view output by SECOND is as follows: Figure 4 shown.

[0077] like Figure 5As shown in the figure, the domain adapter network includes a 3×3 convolution block, a 1×1 convolution block, a 1×1 convolution layer, a downsampling layer and an upsampling layer. The input of the domain adapter network is used as the input of the first 3×3 convolution block. The first 3×3 convolution block is sequentially connected to the first 1×1 convolution block, the downsampling layer, the second 3×3 convolution block, the second 1×1 convolution block, the upsampling layer and the third 3×3 convolution block, and then connected to the third 1×1 convolution block. The output of the first 1×1 convolution block is concatenated with the output of the third 1×1 convolution block and then input to the first 1×1 convolution layer. The output of the first 1×1 convolution layer is used as the output of the domain adapter network. Each 3×3 convolution block includes a 3×3 convolution layer, a batch normalization layer and a RELU activation function layer connected in sequence; each 1×1 convolution block includes a 1×1 convolution layer, a batch normalization layer and a RELU activation function layer connected in sequence.

[0078] 4) Freeze all feature encoders and the network parameters of the V2X fusion backbone network and the detection head, combine the sensor internal and external calibration parameters between the cooperative vehicle and the self-vehicle, and the sensor data corresponding to the self-vehicle and all cooperative vehicles, train each domain adapter network until the training is completed, obtain all trained domain adapter networks and update the second V2X cooperative perception network;

[0079] 4) Specifically:

[0080] 4.1) In each training, the projection matrix and affine transformation matrix between each cooperative vehicle and the self-vehicle are calculated based on the calibration parameters of the sensor internal and external parameters between each cooperative vehicle and the self-vehicle;

[0081] 4.2) Use the collaborative annotation truth value of the collaborative vehicle and each ego vehicle, the single vehicle annotation truth value, and the affine transformation matrix to filter out the common area, that is, the foreground and background that can be observed by both the collaborative vehicle and the ego vehicle, and the rest are non-public areas. Set the public area to 1 and the non-public area to 0, and generate a mask corresponding to each ego vehicle;

[0082] 4.3) Freeze all feature encoders and the network parameters of the V2X fusion backbone network and the detection head, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders of the second V2X cooperative perception network respectively, and then use the projection matrix of each cooperative vehicle to project the feature map outputted by the corresponding feature encoder from a bird's-eye view to the ego vehicle coordinate system to obtain the corresponding projected feature map; then, dot product the projected feature map corresponding to each cooperative vehicle with the mask to obtain the corresponding feature map that only retains the common area, dot product the feature map outputted by the ego vehicle's feature encoder from a bird's-eye view with the mask to obtain the corresponding feature map that only retains the common area and use it as the true value of the common area of ​​the feature map; finally, calculate the mean square error loss of the common area of ​​the feature map of the ego vehicle and the cooperative vehicle based on the true value of the common area of ​​the feature map and the feature map that only retains the common area corresponding to each cooperative vehicle, and use it together with the classification loss, regression loss and direction loss of the detection head to form the total loss, that is, train the domain adapter in a way guided by the final detection result to complete one training;

[0083] 4.4) Repeat 4.1)-4.3) to train each domain adapter network until the training is completed, obtain the trained domain adapter network and update the second V2X collaborative perception network.

[0084] 5) Based on the feature maps collected and extracted by all cooperative vehicles in step 3) and the sensor data collected by the self-vehicle, the current second V2X cooperative perception network is used to perform vehicle cooperative perception to obtain vehicle cooperative perception results.

[0085] The present invention also proposes a V2X cooperative perception system based on a lightweight domain adapter, the system comprising:

[0086] A first training unit is used to train a V2X fusion backbone network and a detection head in a first V2X collaborative perception network to obtain corresponding trained network parameters;

[0087] A second training unit is used to construct a second V2X collaborative perception network based on the trained network parameters corresponding to the V2X fusion backbone network and the detection head, and train the domain adapter network in the second V2X collaborative perception network to obtain a trained second V2X collaborative perception network;

[0088] The vehicle cooperative perception output unit is used to input the feature maps collected and extracted by all cooperative vehicles and the sensor data collected by the self-vehicle into the trained second V2X cooperative perception network, and output the vehicle cooperative perception results.

[0089] Figure 7 The collaborative 3D target detection effect of the present invention on the OPV2V dataset is demonstrated. Figure 7(a) is the basic collaborative detection network of the ego vehicle, namely the PointPillars feature encoder, the V2X fusion backbone network Cobevt and the 1×1Conv detection head. The collaborative 3D target detection result after adding the heterogeneous feature map extracted by the SECOND feature encoder is detected without any processing. Figure 7 (b) shows the collaborative 3D object detection result after adding the heterogeneous feature map extracted by the SECOND feature encoder into the basic collaborative detection network with a non-end-to-end trained domain adapter. Figure 7 (c) shows the collaborative 3D object detection result after inserting the heterogeneous feature map extracted by the SECOND feature encoder into the basic collaborative detection network with the end-to-end trained domain adapter. From the visualization results, it can be seen that without the domain adapter, the heterogeneous feature map causes 4 targets to be undetected in this scene. After inserting the non-end-to-end trained domain adapter, only 2 targets are undetected in this scene. After inserting the end-to-end trained domain adapter, all targets in this scene are detected and the position parameters of the detection box are predicted more accurately.

[0090] Tables 1 and 2 list the quantitative results of 3D target detection based on more V2X collaborative fusion backbone networks under the OPV2V dataset. Three existing V2X collaborative fusion backbone networks were selected. When heterogeneous feature maps were added to the input, collaborative detection reasoning was performed on the networks without domain adapters, with domain adapters trained non-end-to-end, and with domain adapters trained end-to-end. As can be seen from Table 1, the average precision (AP) value of collaborative 3D target detection was greatly improved after inserting domain adapters trained non-end-to-end or end-to-end before these V2X collaborative fusion backbone networks. Among them, the end-to-end training domain adapter can make the average precision (AP) value increase the most.

[0091] Table 1 shows the collaborative 3D target detection results of the method in Example 1 of the present invention and the existing method without domain adapter on the OPV2V dataset

[0092]

[0093] Table 2 shows the collaborative 3D target detection results of the method in Example 2 of the present invention and the existing method without domain adapter on the OPV2V dataset

[0094]

[0095] To further demonstrate the effectiveness of the present invention, Table 3 lists the collaborative 3D target detection results after inserting an end-to-end trained domain adapter in front of the V2X fusion backbone network on another test dataset of OPV2V (the number of cooperative vehicles is at least 3), with different heterogeneous feature map combinations of PointPillars, SECOND (①) and PointPillars, Efficienetb1 (②) and PointPillars, SECOND, Efficienetb1 (③) as input.

[0096] Table 3 shows the collaborative 3D target detection results of the method in Example 2 of the present invention and the existing method without domain adapter on the OPV2V dataset

[0097]

[0098] Finally, it is to be noted that the above embodiments and explanations are only used to illustrate the technical solution of the present invention rather than to limit it. It should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope disclosed in the technical solution of the present invention, which should be included in the scope of protection of the claims of the present invention.

Claims

1. A V2X collaborative perception method based on a lightweight domain adapter, characterized in that: The steps include: 1) The first V2X cooperative perception network is installed in the ego vehicle, and the first V2X cooperative perception network includes a feature encoder of the ego vehicle, a V2X fusion backbone network and a detection head. The feature encoders installed in multiple cooperative vehicles are the same as the feature encoder of the ego vehicle. The feature maps output by the feature encoders of all cooperative vehicles are sent to the ego vehicle and spliced ​​with the output of the feature encoder of the ego vehicle and then input into the V2X fusion backbone network. The V2X fusion backbone network is connected to the detection head; 2) Freeze the network parameters of all feature encoders, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders, and train the V2X fusion backbone network and the detection head until the training is completed, obtain the trained V2X fusion backbone network and the detection head, and update the first V2X cooperative perception network; 3) Adding a corresponding domain adapter network to the feature map input branch of each cooperative vehicle of the current first V2X cooperative perception network to form a second V2X cooperative perception network; for each cooperative vehicle, changing its sensor type and replacing the feature encoder at the same time, or changing only the feature encoder of the cooperative vehicle without changing its sensor type, so that the feature encoders of all cooperative vehicles are different from the feature encoder of the self vehicle; 4) Freeze all feature encoders and the network parameters of the V2X fusion backbone network and the detection head, combine the sensor internal and external calibration parameters between the cooperative vehicle and the self-vehicle, and the sensor data corresponding to the self-vehicle and all cooperative vehicles, train each domain adapter network until the training is completed, obtain all trained domain adapter networks and update the second V2X cooperative perception network; 5) Based on the feature maps collected and extracted by all cooperative vehicles in step 3) and the sensor data collected by the self-vehicle, the current second V2X cooperative perception network is used to perform vehicle cooperative perception to obtain vehicle cooperative perception results.

2. According to claim 1, a V2X collaborative perception method based on a lightweight domain adapter is characterized in that: The feature encoder of the ego vehicle and the feature encoder of the cooperative vehicle are pre-trained feature encoders carried by the ego vehicle / cooperative vehicle itself, or are obtained after the ego vehicle / cooperative vehicle trains its own feature encoder using its own sensor data.

3. The V2X collaborative sensing method based on a lightweight domain adapter according to claim 1, characterized in that: The specific aspects of 4) are: 4.1) In each training, the projection matrix and affine transformation matrix between each cooperative vehicle and the self-vehicle are calculated based on the calibration parameters of the sensor internal and external parameters between each cooperative vehicle and the self-vehicle; 4.2) Use the collaborative annotation truth values ​​of the collaborative vehicle and each ego vehicle, the single vehicle annotation truth value, and the affine transformation matrix to filter out the common area and generate a mask corresponding to each ego vehicle; 4.3) Freeze the network parameters of all feature encoders, V2X fusion backbone network, and detection head, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders respectively, and then use the projection matrix of each cooperative vehicle to project the feature map under the bird's-eye view output of the corresponding feature encoder to the ego vehicle coordinate system to obtain the corresponding projected feature map; Then, the projected feature map corresponding to each cooperative vehicle is dot-producted with the mask to obtain the corresponding feature map that only retains the common area. The feature map output by the feature encoder of the ego vehicle from a bird's-eye view is dot-producted with the mask to obtain the corresponding feature map that only retains the common area and use it as the true value of the common area of ​​the feature map. Finally, the mean square error loss of the common area of ​​the feature map of the ego vehicle and the cooperative vehicle is calculated based on the true value of the common area of ​​the feature map and the feature map of only the common area of ​​each cooperative vehicle and used as the total loss to complete one training; 4.4) Repeat 4.1)-4.3) to train each domain adapter network until the training is completed, obtain the trained domain adapter network and update the second V2X collaborative perception network.

4. The V2X collaborative sensing method based on a lightweight domain adapter according to claim 1, characterized in that: The specific aspects of 4) are: 4.1) In each training, the projection matrix and affine transformation matrix between each cooperative vehicle and the self-vehicle are calculated based on the calibration parameters of the sensor internal and external parameters between each cooperative vehicle and the self-vehicle; 4.2) Use the collaborative annotation truth values ​​of the collaborative vehicle and each ego vehicle, the single vehicle annotation truth value, and the affine transformation matrix to filter out the common area and generate a mask corresponding to each ego vehicle; 4.3) Freeze the network parameters of all feature encoders, V2X fusion backbone network, and detection head, input the sensor data corresponding to the ego vehicle and all cooperative vehicles into the corresponding feature encoders respectively, and then use the projection matrix of each cooperative vehicle to project the feature map under the bird's-eye view output of the corresponding domain adapter network to the ego vehicle coordinate system to obtain the corresponding projected feature map; Then, the projected feature map corresponding to each cooperative vehicle is dot-producted with the mask to obtain the corresponding feature map that only retains the common area. The feature map output by the feature encoder of the ego vehicle from a bird's-eye view is dot-producted with the mask to obtain the corresponding feature map that only retains the common area and use it as the true value of the common area of ​​the feature map. Finally, the mean square error loss of the common area of ​​the feature map of the ego vehicle and the cooperative vehicle is calculated based on the true value of the common area of ​​the feature map and the feature map that only retains the common area of ​​each cooperative vehicle, and the total loss is formed together with the classification loss, regression loss and direction loss of the detection head to complete one training. 4.4) Repeat 4.1)-4.3) to train each domain adapter network until the training is completed, obtain the trained domain adapter network and update the second V2X collaborative perception network.

5. The V2X collaborative sensing method based on a lightweight domain adapter according to claim 1, characterized in that: The domain adapter network includes a 3×3 convolution block, a 1×1 convolution block, a 1×1 convolution layer, a downsampling layer and an upsampling layer. The input of the domain adapter network is used as the input of the first 3×3 convolution block. The first 3×3 convolution block is sequentially connected to the first 1×1 convolution block, the downsampling layer, the second 3×3 convolution block, the second 1×1 convolution block, the upsampling layer and the third 3×3 convolution block, and then connected to the third 1×1 convolution block. The output of the first 1×1 convolution block is spliced ​​with the output of the third 1×1 convolution block and then input into the first 1×1 convolution layer. The output of the first 1×1 convolution layer is used as the output of the domain adapter network.

6. A V2X collaborative sensing method based on a lightweight domain adapter according to claim 5, characterized in that: The 3×3 convolution block includes a 3×3 convolution layer, a batch normalization layer, and a RELU activation function layer connected in sequence; the 1×1 convolution block includes a 1×1 convolution layer, a batch normalization layer, and a RELU activation function layer connected in sequence.

7. A V2X cooperative perception system based on a lightweight domain adapter, characterized in that: include: A first training unit is used to train a V2X fusion backbone network and a detection head in a first V2X collaborative perception network to obtain corresponding trained network parameters; A second training unit is used to construct a second V2X collaborative perception network based on the V2X fusion backbone network and the trained network parameters corresponding to the detection head, and train the domain adapter network in the second V2X collaborative perception network to obtain a trained second V2X collaborative perception network; The vehicle collaborative perception output unit is used to input the sensor data collected by the self-vehicle and all collaborative vehicles into the trained second V2X collaborative perception network respectively, and output the vehicle collaborative perception results.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the V2X collaborative perception method based on a lightweight domain adapter as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the V2X collaborative perception method based on a lightweight domain adapter as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the V2X collaborative perception method based on a lightweight domain adapter as described in any one of claims 1 to 6 are implemented.

Images