A laser radar-based heterogeneous cooperative perception 3D target detection method and device

Abstract

The present application relates to the technical field of cooperative perception, and particularly relates to a heterogeneous cooperative perception 3D target detection method and device based on laser radar. The method comprises: a two-stage heterogeneous cooperative perception training framework, the two stages comprising a local training stage and a cooperative training stage; a minimum difference method between intelligent agents is designed to fuse features between the intelligent agents participating in cooperation; a global view assisted reinforcement method is designed, global feature maps and single view feature maps are subjected to gradient inversion layers and combined with domain classification heads for domain adversarial training, global information is used to enhance the feature representation capability of each intelligent agent, and a unified feature space is constructed. The method provided by the present application can independently update the fusion network of the cooperative perception task in a complex driving scene without retraining each backbone network, while keeping the performance of single vehicle perception unaffected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cooperative perception, in particular to a heterogeneous cooperative perception 3D target detection method and device based on laser radar. BACKGROUND

[0002] Autonomous vehicles are widely recognized as an effective means to improve road safety. However, the inherent limitations of single-vehicle perception systems, such as susceptibility to occlusions, limited sensor coverage, and challenges in long-range perception, make them face many problems. In recent years, in response to these problems in single-vehicle perception, multi-agent cooperative perception technology has been developed, such as vehicle-to-vehicle (V2V) and vehicle-to-everything (V2X) cooperation. In these systems, multiple agents located at different positions in the same environment cooperate and exchange information through communication, thereby constructing a unified global perception graph and improving the performance of single-vehicle perception. However, most existing researches focus on cooperative systems based on homogeneous agents. These methods require the same type of agents, thereby limiting their applicability and flexibility in real-world scenarios.

[0003] To facilitate real-world applications, several methods have studied the implementation of heterogeneous cooperative perception by relaxing the same data type constraint and using data from different sensors, such as mechanical laser radars with different beam numbers or semi-solid laser radars. However, sharing raw data for cooperative training may compromise the privacy of raw data from each company. In addition, these methods use the same backbone network to process all cooperative perception agents. This unified backbone network may not achieve the best perception for different types of laser radar data.

[0004] Recently, the HEAL[Yifan Lu, Yue Hu, Yiqi Zhong, Dequan Wang, Yanfeng Wang, and Siheng Chen. An extensible framework for open heterogeneous collaborative perception.] method not only integrates data from different laser radar sensors, but also sets different backbone network types for each agent and adopts a two-stage training strategy with back-alignment. The HEAL method first trains a homogeneous cooperative perception network using data from a single sensor, then fixes the fusion subnetwork, and trains different backbone network types for different agents. When the fusion network is updated, the backbone networks of all other types of agents must be retrained, which will bring additional training costs to other types of agents (possibly from different companies), increase the complexity of the cooperative process, and affect the perception performance of individual agents. SUMMARY

[0005] To solve the technical problem in the prior art that when the fusion network is updated, the backbone network of all other types of agents must be retrained, which will bring additional training cost to other types of agents from different companies, an embodiment of the present application provides a laser radar based heterogeneous collaborative perception 3D target detection method and device. The technical solution is as follows:

[0006] In one aspect, a laser radar based heterogeneous collaborative perception 3D target detection method is provided, characterized in that the method comprises:

[0007] S1, a two-stage heterogeneous collaborative perception training framework is constructed, the two stages including a local training stage and a collaborative training stage, and the agents participating in collaboration are independently trained and updated through the local training stage; wherein the agents participating in collaboration adopt different line number laser radar sensors and different network backbone structures;

[0008] S2, the agents participating in collaboration are aligned in features through the collaborative training stage; a method of minimizing the difference between agents is designed to reduce the feature map difference between agents;

[0009] S3, a global view assisted reinforcement method is designed, the global feature map and the single view feature map are passed through a gradient inversion layer and combined with a domain classification head for domain adversarial training, the global information is used to enhance the feature representation ability of each agent, a unified feature space is constructed, and laser radar based heterogeneous collaborative perception 3D target detection is completed.

[0010] Optionally, the independent training and update of the agents participating in collaboration through the local training stage comprises:

[0011] The backbone corresponding to each agent type independently trains the single vehicle perception model of the agent through the local training stage of the heterogeneous collaborative perception training framework;

[0012] Wherein, the backbone corresponding to the agent type only uses its own data when independently training and updating, and does not share the original data when collaborative training;

[0013] When a new agent type needs to be added, the feature map F i =Φ i (X i )∈R (H×W×C) , wherein Φ i represents the corresponding backbone network, H, W and C represent the height, width and channel number of the feature map respectively; the single vehicle perception model of the agent is independently trained and updated through the local training stage of the two-stage heterogeneous collaborative perception training framework.

[0014] Optionally, in S2, feature alignment is performed on the participating agents in the cooperative training phase, including:

[0015] In the cooperative training phase of the heterogeneous cooperative perception training framework, the models obtained by the participating cooperative agents in the first phase are loaded, and the BEV feature maps of each agent are obtained by inputting the data required for cooperative training;

[0016] Then, through the preliminary scale alignment layer, the features from different agents are preliminarily aligned using 1x1 convolution;

[0017] In the cooperative training phase of the heterogeneous cooperative perception training framework, the parameter information of all participating cooperative agents is aligned with the ego vehicle in terms of perspective;

[0018] The feature alignment includes: the participating cooperative vehicles communicate the coordinate information to the ego vehicle, and the ego vehicle calculates the coordinate transformation matrix according to the coordinate information of the participating cooperative vehicles to convert the feature maps of the participating cooperative vehicles to the coordinate system of the ego vehicle;

[0019] The transformed feature is represented as Wherein represents the feature transformation operation.

[0020] Optionally, in S2, a method of minimizing the difference between agents is designed to reduce the difference between the feature maps of the agents, including:

[0021] The feature maps of each agent are represented as {F′1, F′2,... F′ i , F′ j F′ k ,... F ′ N};

[0022] A multi-scale fusion network is constructed, and the feature maps of each agent {F′1, F′2,... F′ i , F′ j F′ k ,... F′ N} are input into the multi-scale fusion network; the feature maps extracted at each scale are represented as Wherein α represents the αth scale, i, j, k correspond to the agents participating in cooperative perception;

[0023] The foreground estimation maps S of two agents participating in cooperative perception are calculated; 1x1 convolution is applied to the feature maps of each agent at each scale, and the foreground estimation map of each agent at scale α is calculated as follows:

[0024]

[0025] Based on the foreground estimation map of each agent at scale alpha, the weight matrix between agents i and j is calculated:

[0026]

[0027] At each scale alpha, the KL divergence m between the feature maps of agents i and j is calculated α :

[0028]

[0029] Where N is the total number of collaborative agents, i, j∈{1, 2,..., N}; A is the total number of scales, alpha∈{1, 2,..., A};

[0030] The total difference at all scales is calculated :

[0031]

[0032] Where M represents the total difference loss;

[0033] By minimizing Align the feature maps of different agents at each scale, and perform feature fusion on all participating collaborative agents.

[0034] Optionally, a global view assisted reinforcement method is designed, which combines the global feature map and the single view feature map through the gradient reversal layer and the domain classification head for domain adversarial training, uses global information to enhance the feature representation ability of each agent, and constructs a unified feature space, including:

[0035] Obtain the feature map of different agents at scale alpha

[0036] Input the feature map of different agents at scale alpha into the foreground generator to obtain the foreground estimation map;

[0037] Use the softmax function to normalize the foreground estimation map to generate the weight matrix W for feature fusion;

[0038] Calculate the weighted feature map of different agents at scale alpha:

[0039]

[0040] Merge the fused feature map at scale alpha With the feature map of a single view to form a combined feature set

[0041] The combined feature set Fall is input to the gradient inversion layer to obtain inverted features, and the inverted features are input to the domain classifier to determine whether the input features are global fusion features or single-view features, wherein the fusion features have a label of 1, and the single-view features have a label of 0; and a domain classification result is output. The network structure of the domain classifier is as follows:

[0042] Two 3*3 convolution layers with padding of 1:

[0043] Y1 = relu(Conv 3×3 (F all ))

[0044] Y2 = relu(Conv 3×3 (Y1)

[0045] Maximum value pooling is performed on the input feature map:

[0046] Y3 = maxpool(Y2)

[0047] The feature map is flattened:

[0048] Y3 = flatten(Y2)

[0049] After transformation by three fully connected layers FC, the original output logits without an activation function are obtained.

[0050] Y5 = dropout(relu(fc(Y)))

[0051] Y6 = dropout(relu(fc(Y5)))

[0052] logits = fc(Y6)

[0053] Based on the domain classification result, the domain classification cross-entropy loss at each scale a and the total domain classification loss at all scales are calculated.

[0054] Optionally, the domain classification cross-entropy loss at each scale a and the total domain classification loss at all scales are calculated, including:

[0055] The domain classification cross-entropy loss at each scale a is calculated as follows:

[0056]

[0057] wherein y refers to logits obtained by the domain classifier.

[0058] The field classification cross-entropy loss on each scale a is calculated:

[0059]

[0060] Optionally, S3 further comprises:

[0061] The overall loss function of the training model is calculated:

[0062]

[0063] Wherein, β, γ and ∈ are hyperparameters.

[0064] In another aspect, a laser radar-based heterogeneous collaborative perception 3D target detection device is provided, which is applied to a laser radar-based heterogeneous collaborative perception 3D target detection method, and the device comprises: a data preprocessing module, configured to input a preset input program to a compiler front end and output an intermediate representation;

[0065] A heterogeneous collaborative perception framework design module is configured to construct a heterogeneous collaborative perception training framework; each manufacturer type backbone updates an intelligent agent independently through the heterogeneous collaborative perception training framework.

[0066] A consistency enhancement module is configured to fuse features through the heterogeneous collaborative perception training framework after obtaining BEV feature maps of participating collaborative vehicles; the consistency enhancement module comprises: a minimum inter-agent feature difference submodule and a global perspective auxiliary enhancement submodule.

[0067] The minimum inter-agent feature difference submodule is configured to design a minimum inter-agent difference method to reduce the difference between feature maps of intelligent agents.

[0068] The global perspective auxiliary enhancement submodule is configured to design a global view auxiliary reinforcement method, which performs domain adversarial training on global feature maps and single-view feature maps through a gradient inversion layer and a domain classification head, enhances the feature representation capability of each intelligent agent by using global information, constructs a unified feature space, and completes laser radar-based heterogeneous collaborative perception 3D target detection.

[0069] In another aspect, a laser radar-based heterogeneous collaborative perception 3D target detection device is provided, which comprises: a processor; and a memory having computer readable instructions stored thereon, wherein the computer readable instructions are executed by the processor to implement any one of the above laser radar-based heterogeneous collaborative perception 3D target detection methods.

[0070] In another aspect, a computer-readable storage medium is provided, the storage medium having stored therein at least one instruction, the at least one instruction being loaded and executed by a processor to implement any one of the above laser radar-based heterogeneous collaborative perception 3D target detection methods.

[0071] The technical scheme provided by the embodiment of the present application has at least the following beneficial effects:

[0072] In the embodiment of the present application, the core method of the framework mainly includes three parts: 1) a two-stage training strategy for a laser radar-based heterogeneous collaborative perception task is proposed, so that each agent of the collaborative perception system of each manufacturer remains relatively independent, the data of each agent has a certain confidentiality, and the single vehicle perception performance of each agent is not affected by the collaborative perception task and is not affected by the feature fusion network update; 2) in the feature fusion module part, an inter-agent difference minimization design is introduced to reduce the difference between the features of different perspective heterogeneous agents in the same scene and improve the feature fusion effect; 3) in the feature fusion module part, a global view auxiliary reinforcement design is introduced, the global feature map and the single perspective feature map are passed through a gradient inversion layer and combined with a domain classification head for domain adversarial training, the global information is used to enhance the feature representation ability of each agent, a unified feature space is constructed, and the domain difference is reduced, which ensures the effectiveness and stability of multi-perspective feature fusion and enhances the robustness of collaborative perception. A large number of experiments show that for V2X and V2V heterogeneous collaborative perception, the perception effect of the present application is better than that of the existing most advanced method. BRIEF DESCRIPTION OF DRAWINGS

[0073] In order to more clearly illustrate the technical scheme in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0074] Figure 1 A flowchart of a laser radar-based heterogeneous collaborative perception 3D target detection method provided by the embodiment of the present application is shown in the figure.

[0075] Figure 2 A laser radar-based heterogeneous collaborative perception 3D target detection map provided by the embodiment of the present application is shown in the figure.

[0076] Figure 3 A block diagram of a laser radar-based heterogeneous collaborative perception 3D target detection device provided by the embodiment of the present application is shown in the figure.

[0077] Figure 4 A structural schematic diagram of an electronic device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0078] The technical solutions in the present application will be described below with reference to the drawings.

[0079] In the embodiments of the present application, the words such as "exemplary", "for example", etc. are used to represent an example, illustration or description. Any embodiment or design scheme described as "exemplary" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "exemplary" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be either one of the two.

[0080] In the embodiments of the present application, sometimes the subscript such as W1 may be written in the form of non-subscript such as W1, and when the difference is not emphasized, the meanings expressed are consistent.

[0081] In order to make the technical problems, technical solutions and advantages of the present application clearer, the following will be described in detail with reference to the drawings and specific embodiments.

[0082] The embodiments of the present application provide a heterogeneous collaborative perception 3D target detection method based on laser radar, which can be implemented by a heterogeneous collaborative perception 3D target detection device based on laser radar. The device can be a terminal or a server. As shown in the heterogeneous collaborative perception 3D target detection method based on laser radar, the method can include the following steps: Figure 1 As shown in the flowchart of the heterogeneous collaborative perception 3D target detection method based on laser radar, the heterogeneous collaborative perception 3D target detection method based on laser radar provided by the present application can include the following steps: Figure 1 As shown in the flowchart of the heterogeneous collaborative perception 3D target detection method based on laser radar, the heterogeneous collaborative perception 3D target detection method based on laser radar provided by the present application can include the following steps:

[0083] S1, a two-stage heterogeneous collaborative perception training framework is constructed, the two stages include a local training stage and a collaborative training stage, and the intelligent agents participating in collaboration are independently trained and updated through the local training stage; wherein the intelligent agents participating in collaboration adopt different linear laser radar sensors and different network backbone structures.

[0084] In a feasible implementation, the application provides a heterogeneous collaborative perception training framework, which is a two-stage training strategy for a laser radar-based heterogeneous collaborative perception task. The original data of each type of agent has privacy protection, and the integration of different types of collaborative agents can be realized. When a new agent is added, the collaborative network can be updated independently without affecting other participating collaborative agents, and there is no need to retrain their private backbone networks to maximize the adaptability of heterogeneous features extracted from the LiDAR sensors of different agents. In practical applications, the best single-vehicle perception model of each manufacturer is independently trained and has privacy, so it is unrealistic to require each manufacturer to contribute its data for collaborative perception task training. Therefore, the application fixes the backbone network for each manufacturer, retains its best single-vehicle perception performance, and does not affect the original decision-making module of each vehicle.

[0085] In a feasible implementation, in S1, the participating collaborative agents are independently trained and updated through the local training stage, including:

[0086] The backbone corresponding to each agent type independently trains the single-vehicle perception model of the agent through the local training stage of the heterogeneous collaborative perception training framework.

[0087] Wherein, the backbone corresponding to the agent type only uses its own data when independently training and updating, and does not share the original data during collaborative training.

[0088] When a new agent type needs to be added, the feature map F i extracted by the backbone network corresponding to the new agent type is added to the collaborative network. i i ∈R (H×W×C) , wherein Φ i represents the corresponding backbone network, H, W and C represent the height, width and channel number of the feature map, respectively.

[0089] In a feasible implementation, in the local training stage, each type of agent independently trains its single-vehicle perception model. As shown on the left side, Figure 2 each agent converts the original 3D point cloud into a bird's eye view (BEV) map X i . The feature map F i extracted by its corresponding backbone network is i i ∈R (H×W×C) , wherein Φ i represents the corresponding backbone network, H, W and C represent the height, width and channel number of the feature map, respectively. ​​

[0090] In the embodiment of the present application, the new heterogeneous training framework has the characteristics of protecting the privacy backbone network, so that in complex driving scenarios, the fusion network of collaborative perception tasks can be updated independently without retraining each backbone network, while maintaining the optimal performance of single vehicle perception without being affected.

[0091] In the collaborative training phase, the agents participating in collaboration are aligned in feature, and a method of minimizing the difference between agents is designed to reduce the difference between feature maps of agents.

[0092] In a feasible implementation, the feature alignment of the agents participating in collaboration in the collaborative training phase includes:

[0093] In the collaborative training phase of the heterogeneous collaborative perception training framework, the models obtained by the first phase training of all participating collaborative agents are loaded, the required data for collaborative training is input, and the BEV feature map of each agent is obtained through the backbone of each agent type;

[0094] Then, through the preliminary scale alignment layer, the feature maps from different agents are preliminarily aligned by using 1x1 convolution;

[0095] In the collaborative training phase of the heterogeneous collaborative perception training framework, the parameter information of all participating collaborative agents is converted and aligned with the perspective of the ego vehicle;

[0096] The feature alignment includes that the participating collaborative vehicles communicate the coordinate information to the ego vehicle, and the ego vehicle calculates the coordinate conversion matrix according to the coordinate information of the participating collaborative vehicles to convert the feature maps of the participating collaborative vehicles to the coordinate system of the ego vehicle.

[0097] The transformed feature is represented as Wherein represents the feature transformation operation.

[0098] In the collaborative training phase, the backbone network parameters of each agent type obtained in the local training are fixed. As shown in Figure 2 , one vehicle is called ego vehicle, which broadcasts its own position information to all other collaborative vehicles. After receiving the position information of the ego vehicle, each collaborative vehicle will perform perspective transformation on its feature map F i , so that it is aligned with the feature of the ego vehicle. The transformed feature is represented as Wherein represents the feature transformation operation.

[0099] In a feasible implementation, in S2, a method of minimizing the difference between agents is designed to reduce the difference between feature maps of agents, including:

[0100] {F'1, F'2,... F'N} of each agent are input into the multi-scale fusion network, and the feature map representation extracted at each scale is obtained i j k N

[0101] {F'1, F'2,... F'N} of each agent are input into the multi-scale fusion network, and the feature map representation extracted at each scale is obtained i j k N wherein α represents the αth scale, i, j, k correspond to agents participating in cooperative perception;

[0102] The foreground estimation map S of two agents participating in cooperative perception is calculated; a 1x1 convolution is respectively applied to the feature map of each agent at each scale, and the foreground estimation map of each agent at scale α is calculated as follows:

[0103]

[0104] Based on the foreground estimation map of each agent at scale α, the weight matrix between agents i and j is calculated:

[0105]

[0106] At each scale α, the KL divergence m α between the feature maps of agents i and j is calculated:

[0107]

[0108] wherein N is the total number of cooperative agents, i, j ∈ {1, 2,..., N}; A is the total number of scales, α ∈ {1, 2,..., A};

[0109] The total difference at all scales is calculated as:

[0110]

[0111] wherein M represents the total difference loss;

[0112] By minimizing the feature maps of different agents are aligned at each scale, and the feature fusion is performed for all participating cooperative agents.

[0113] ​​​​​​​​In one feasible implementation, the backbone network parameters learned during training may differ. These differences directly affect the feature maps {F′1, F′2, ... F′} extracted from each agent. i F′ j F′ k ,...F′ N The differences between feature maps of different agents lead to changes in information distribution. These differences make subsequent feature fusion more complex and increase the difficulty of feature alignment and sharing. To minimize the differences in feature maps between agents, a simple and effective method is to use KL divergence to measure the differences in feature distribution. Therefore, this invention calculates the KL divergence between feature maps of different agents and uses the foreground estimation map of the agents to calculate the loss weights, reducing meaningless alignments and increasing the weight of effective alignments.

[0114] In one possible implementation, in this general form, The overall differences between all scales and agents were quantified. This was achieved by minimizing... This invention aligns feature maps of different agents at each scale, ensuring similar feature distributions at each spatial location. This operation improves feature alignment quality across multiple spatial scales, produces more robust multi-scale feature representations, and ensures that these features remain consistent across different agents.

[0115] S3. Design a global view-assisted enhancement method, which passes the global feature map and single-view feature map through a gradient inversion layer and combines them with a domain classification head for domain adversarial training. It utilizes global information to enhance the feature representation ability of each agent, constructs a unified feature space, and completes heterogeneous collaborative perception 3D target detection based on LiDAR.

[0116] In one feasible implementation, in S3, a global view-assisted enhancement method is designed. The global feature map and the single-view feature map are passed through a gradient inversion layer and combined with a domain classification head for domain adversarial training. Global information is used to enhance the feature representation capabilities of each agent, constructing a unified feature space, including:

[0117] Obtain feature maps of different agents at scale α

[0118] The feature maps of different agents at scale α are input into the foreground generator to obtain the foreground estimation map;

[0119] The foreground estimation map is normalized using the softmax function to generate a weight matrix W for feature fusion.

[0120] Calculate the weighted feature maps of different agents at scale α:

[0121]

[0122] the feature map after fusion at scale a merge with the feature map of the single view to form a combined feature set

[0123] the combined feature set Fall is input into the gradient inversion layer to obtain the inverted feature, and the inverted feature is input into the domain classifier, and it is determined by the domain classifier whether the input feature is from global fusion feature or single view feature, wherein the fusion feature is set to 1, and the single view feature is set to 0; output the domain classification result;

[0124] wherein the network structure of the domain classifier is as follows:

[0125] two layers of 3*3 convolution layers with padding of 1:

[0126] Y1=relu(Conv 3×3 (F all ))

[0127] Y2=relu(Conv 3×3 (Y1))

[0128] maximum pooling is performed on the input feature map:

[0129] Y3=maxpool(Y2)

[0130] the feature map is flattened:

[0131] Y3=flatten(Y2)

[0132] after transformation through three fully connected layers FC, the original output logits without activation function is finally obtained;

[0133] Y5=dropout(relu(fc(Y4)))

[0134] Y6=dropout(relu(fc(Y5)))

[0135] logits=fc(Y6)

[0136] Based on the domain classification result, the domain classification cross-entropy loss at each scale a and the total domain classification loss at all scales are calculated.

[0137] In a feasible implementation, although the differences between the agents have been minimized, it is still necessary to enhance the features of each agent using the overall information. To solve this problem, the invention can use the fused features (Fu ) to form the overall information guiding the alignment between different agents. However, due to the occlusion problem, there are many empty regions in the feature map, and the regions observed by different agents are also different. When using KL divergence for alignment, the empty features may produce meaningless alignment with the non-empty features, which reduces the expression ability of the network. Therefore, the present application selects the field adversarial training, which uses adversarial training as a kind of soft alignment technology.

[0138] In an implementable embodiment, the field classification cross-entropy loss on each scale a and the total field classification loss on all scales are calculated, including:

[0139] The field classification cross-entropy loss on each scale a is calculated as:

[0140]

[0141] where y refers to the logits obtained by the domain classifier;

[0142] The field classification cross-entropy loss on each scale a is calculated as:

[0143]

[0144] In an implementable embodiment, the main goal of the domain classifier is to distinguish whether the input features are from the global fusion features or from the single-view features, so that the model can effectively distinguish the features from different sources. The gradient reversal layer L is introduced into the adversarial learning process between the feature extractor and the domain classifier, so that the feature extractor learns a way to make the features from different domains difficult to distinguish. This encourages the model to suppress the domain-specific feature differences during the learning process, and promotes the extraction of cross-domain adaptive features.

[0145] Therefore, the model gradually balances the relationship between global features and local features, produces feature representations with consistency under different views, and reduces domain differences. This design ensures the rationality and stability of multi-view feature fusion, and enhances the robustness of collaborative perception.

[0146] In an implementable embodiment, S3 further includes:

[0147] The fused feature map F u is input to the classification head (for classifying foreground / background categories) and the regression head (for generating bounding boxes). The perception result on the ego vehicle is represented as Y = head(F u ), where head represents the classification head and the regression head. In addition to the two detection losses L cls and L reg , the present application also uses the cross-entropy loss L dato supervise the field adversarial training and use the inter-agent difference minimization loss L M to supervise the multi-scale feature extraction process. The overall loss function of the training model is calculated as follows:

[0148]

[0149] where β, γ and ∈ are hyperparameters. In the first stage (local training stage), the present application adopts L cls + βL reg for training. In the second stage (collaborative training stage), the present application uses the total loss in the formula to train the model.

[0150] In the embodiment of the present application, a new two-stage method, double-consistency multi-agent collaborative perception, is proposed. A heterogeneous training framework with a privacy-protecting backbone network is adopted, and includes a local training stage and a collaborative training stage. In the local training stage, a single-vehicle perception model is trained for each type of collaborative vehicle. In order to ensure that each agent achieves the best single-vehicle perception performance under its corresponding model and to ensure the privacy of the data, the present application strictly assigns different types of data to the corresponding models for training. In the collaborative training stage, the backbone network of the agent remains fixed, and only the fusion network and the subsequent detection head are trained.

[0151] Figure 3 is a block diagram of a laser radar-based heterogeneous collaborative perception 3D target detection device 300 for a laser radar-based heterogeneous collaborative perception 3D target detection method according to an exemplary embodiment. Referring to Figure 3 , the device includes a heterogeneous collaborative perception framework design module 310, a consistency enhancement module 320, a minimum inter-agent feature difference submodule 321, and a global perspective auxiliary enhancement submodule 322. Among them:

[0152] The heterogeneous collaborative perception framework design module 310 is used to construct a heterogeneous collaborative perception training framework; each manufacturer type backbone updates the agent independently through the heterogeneous collaborative perception training framework;

[0153] The consistency enhancement module 320 is used to obtain the BEV feature map after participating in the collaborative vehicle, and then perform feature fusion through the heterogeneous collaborative perception training framework; the consistency enhancement module 320 includes: the minimum inter-agent feature difference submodule 321 and the global perspective auxiliary enhancement submodule 322;

[0154] The minimum inter-agent feature difference submodule 321 is used to design a minimum inter-agent difference method to reduce the feature map difference between agents;

[0155] The global view auxiliary enhancement module 322 is used to design a global view auxiliary reinforcement method. The global feature map and the single-view feature map are subjected to gradient inversion layer and combined with the domain classification head for domain adversarial training. The global information is used to enhance the feature representation capability of each agent. A unified feature space is constructed. The heterogeneous collaborative perception 3D target detection based on the laser radar is completed.

[0156] Optionally, the agents participating in the collaboration are independently trained and updated through the local training stage, including:

[0157] The backbone corresponding to each agent type independently trains the single-vehicle perception model of the agent through the local training stage of the heterogeneous collaborative perception training framework.

[0158] When the backbone corresponding to the agent type is independently trained and updated, only its own data is used, and the original data is not shared in the collaborative training.

[0159] When a new agent type needs to be added, the feature map F i = Φ i (X i ) ∈ R (H×W×C) , wherein Φ i represents the corresponding backbone network, H, W and C represent the height, width and channel number of the feature map, respectively. The single-vehicle perception model of the agent is independently trained and updated through the local training stage of the two-stage heterogeneous collaborative perception training framework.

[0160] Optionally, the minimum feature difference between agents submodule 321 is used to:

[0161] Through the collaborative training stage of the heterogeneous collaborative perception training framework, the models obtained by the agents participating in the collaboration in the first stage are loaded. The BEV feature map of each agent is obtained by inputting the data required for collaborative training.

[0162] Then, the feature maps from different agents are preliminarily aligned through a preliminary scale alignment layer using a 1x1 convolution.

[0163] Through the collaborative training stage of the heterogeneous collaborative perception training framework, the parameter information of all the agents participating in the collaboration is aligned with the ego vehicle in terms of perspective.

[0164] The feature alignment includes: the participating collaborative vehicles communicate the coordinate information to the ego vehicle. The ego vehicle calculates the coordinate transformation matrix according to the coordinate information of the participating collaborative vehicles, and converts the feature maps of the participating collaborative vehicles to the coordinate system of the ego vehicle.

[0165] The transformed feature is represented as , wherein represents the feature transformation operation.

[0166] Optionally, the submodule 321 for minimizing feature differences between agents is used for:

[0167] The feature map of each agent {F′1, F′2, ..., F′} i F′ j F′ k ,...F′ N};

[0168] Construct a multi-scale fusion network, combining the feature maps {F′1, F′2, ... F′} of each agent. i F′ j F′ k ,...F′ N The input is fed into a multi-scale fusion network to obtain the feature map representation extracted at each scale. Where α represents the α-th scale, and i, j, and k correspond to the agents participating in collaborative perception;

[0169] Calculate the foreground estimation map S of the two agents participating in collaborative perception; apply 1x1 convolutions to the feature map of each agent at each scale, and calculate the foreground estimation map of each agent at scale α as follows:

[0170]

[0171] Based on the foreground estimation map of each agent at scale α, calculate the weight matrix between agents i and j:

[0172]

[0173] At each scale α, calculate the KL divergence m between the feature maps of agents i and j. α :

[0174]

[0175] Where N is the total number of cooperative agents, i, j∈{1, 2, ..., N}; A is the total number of scales, α∈{1, 2, ..., A};

[0176] Calculate the total difference across all scales for:

[0177]

[0178] Where M represents the total variance loss;

[0179] By minimizing Align the feature maps of different agents at each scale and perform feature fusion on all participating agents.

[0180] Optionally, the global view auxiliary enhancement sub-module 322 is used for:

[0181] obtaining feature maps of different agents at scale a

[0182] inputting the feature maps of different agents at scale a into a foreground generator to obtain a foreground estimation map;

[0183] normalizing the foreground estimation map using a softmax function to generate a weight matrix W for feature fusion;

[0184] calculating the weighted feature maps of different agents at scale a:

[0185]

[0186] merging the fused feature maps at scale a with the feature maps of individual views to form a combined feature set

[0187] inputting the combined feature set Fall into a gradient inversion layer to obtain inverted features, and inputting the inverted features into a domain classifier to determine whether the input features are from global fusion features or individual view features, wherein the label of the fusion features is set to 1, and the label of the individual view features is set to 0; outputting the domain classification result;

[0188] wherein the network structure of the domain classifier is:

[0189] two layers of 3*3 convolution layers with padding of 1:

[0190] Y1 = relu(Conv 3×3 (F all ))

[0191] Y2 = relu(Conv 3×3 (Y1))

[0192] performing maximum pooling on the input feature map:

[0193] Y3 = maxpool(Y2)

[0194] flattening the feature map:

[0195] Y3 = flatten(Y2)

[0196] after passing through three fully connected layers FC, the original output logits without activation function are finally obtained;

[0197] Y5 = dropout(relu(fc(Y4)))

[0198] Y6 = dropout(relu(fc(Y5)))

[0199] logits = fc(Y6)

[0200] Based on the domain classification result, the domain classification cross-entropy loss at each scale a and the total domain classification loss at all scales are calculated.

[0201] Optionally, the global perspective auxiliary enhancer module 322 is configured to:

[0202] Calculate the domain classification cross-entropy loss at each scale a:

[0203]

[0204] Where y refers to the logits obtained by the domain classifier;

[0205] Calculate the domain classification cross-entropy loss at each scale a:

[0206]

[0207] Optionally, the global perspective auxiliary enhancer module 322 is further configured to:

[0208] Calculate the total loss function of the training model:

[0209]

[0210] Where β, γ and ∈ are hyperparameters.

[0211] In the embodiments of the present application, a new two-stage method, double consistency multi-agent collaborative perception (DCon), is proposed. DCon adopts a heterogeneous training framework with a privacy-protecting backbone network, and includes a local training phase and a collaborative training phase. In the local training phase, DCon trains a single-vehicle perception model for each type of collaborative vehicle. In order to ensure that each agent achieves the best single-vehicle perception performance under its corresponding model, and to ensure the privacy of the data, we strictly assign different types of data to the corresponding model for training. In the collaborative training phase, the backbone network of the agent remains fixed, and only the fusion network and the subsequent detection head are trained.

[0212] Figure 4 is a structural schematic diagram of a laser radar-based heterogeneous collaborative perception 3D target detection device provided by the embodiments of the present application, as Figure 4 shown, the laser radar-based heterogeneous collaborative perception 3D target detection device can include the aboveFigure 3 The illustrated device is a heterogeneous collaborative sensing 3D target detection device based on lidar. Optionally, the lidar-based heterogeneous collaborative sensing 3D target detection device 410 may include a first processor 2001.

[0213] Optionally, the heterogeneous collaborative sensing 3D target detection device 410 based on lidar may also include a memory 2002 and a transceiver 2003.

[0214] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0215] The following is combined Figure 4 The following is a detailed introduction to each component of the LiDAR-based heterogeneous collaborative sensing 3D target detection device 410:

[0216] The first processor 2001 is the control center of the heterogeneous collaborative sensing 3D target detection device 410 based on lidar. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0217] Optionally, the first processor 2001 can execute various functions of the lidar-based heterogeneous collaborative sensing 3D target detection device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0218] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0219] In a specific implementation, as one example, the heterogeneous collaborative sensing 3D target detection device 410 based on lidar may also include multiple processors, for example... Figure 4The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0220] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0221] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the lidar-based heterogeneous collaborative sensing 3D target detection device 410. Figure 4 (Not shown in the figure) is coupled to the first processor 2001, and the embodiments of the present invention do not specifically limit this.

[0222] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0223] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0224] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently, and can be connected to the interface circuit of the lidar-based heterogeneous collaborative sensing 3D target detection device 410. Figure 4 (Not shown in the figure) is coupled to the first processor 2001, and the embodiments of the present invention do not specifically limit this.

[0225] It should be noted that, Figure 4 The structure of the laser radar-based heterogeneous collaborative perception 3D target detection device 410 shown in the figure is not a limitation of the router, and the actual knowledge structure recognition device can include more or fewer components than the illustration, or combine certain components, or different component arrangements.

[0226] In addition, the technical effects of the laser radar-based heterogeneous collaborative perception 3D target detection device 410 can refer to the technical effects of the laser radar-based heterogeneous collaborative perception 3D target detection method described in the above method embodiments, which will not be repeated here.

[0227] It should be understood that the first processor 2001 in the embodiments of the present application can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0228] It should also be understood that the memory in the embodiments of the present application can be volatile or nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. The volatile memory can be random access memory (RAM) used as external cache. By way of example, and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0229] The above-described embodiments can be implemented in part or in whole through software, hardware (e.g., circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When loaded and executed by a computer, the computer instructions or computer programs can produce the processes or functions described above in accordance with the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, such as from a website, a computer, a server, or a data center to another website, computer, server, or data center through a wired (e.g., infrared, wireless, microwave, etc.) or wireless channel. The computer-readable storage medium can be any available medium or a collection of medium containing one or more of the available medium that is accessible by a computer. The available medium can be a magnetic medium (e.g., a floppy diskette, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state memory.

[0230] It should be understood that the term "and / or" in this document is merely used to describe associated objects, and can represent three conditions: A and / or B, such as A alone, B alone, or A and B together. In addition, the character " / " in this document generally represents an "or" relationship between the front and rear associated objects, but can also represent an "and / or" relationship. The specific meaning can be understood according to the context before and after.

[0231] It should be understood that in various embodiments of the present application, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0232] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0233] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0234] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit.

[0235] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0236] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for laser-radar-based heterogeneous collaborative perception 3D target detection, characterized in that, The method comprises: S1, constructing a two-stage heterogeneous collaborative perception training framework, the two stages comprising a local training stage and a collaborative training stage, and the intelligent agents participating in collaboration are independently trained and updated through the local training stage; wherein the intelligent agents participating in collaboration adopt different line laser radar sensors and different network backbone structures; S2, aligning the features of the intelligent agents participating in collaboration through the collaborative training stage; designing a method of minimizing the differences between the intelligent agents to reduce the differences between the feature maps of the intelligent agents; S3, designing a global view auxiliary reinforcement method, performing domain adversarial training on the global feature map and the single-view feature map through a gradient reversal layer and combining a domain classification head, using global information to enhance the feature representation capability of each intelligent agent, constructing a unified feature space, and completing the laser radar-based heterogeneous collaborative perception 3D target detection.

2. The laser-radar-based heterogeneous co-perception 3D target detection method according to claim 1, characterized in that, Through the local training stage, the intelligent agents participating in collaboration are independently trained and updated, including: The backbone corresponding to each intelligent agent type independently trains the single-vehicle perception model of the intelligent agent through the local training stage of the heterogeneous collaborative perception training framework; Wherein, the backbone corresponding to the intelligent agent type only uses its own data when independently training and updating, and does not share original data during collaborative training; When a new intelligent agent type needs to be added, the feature map F extracted by the backbone network corresponding to the new intelligent agent type is added i = Φ i (X i ) ∈ R (H×W×C) , wherein Φ i represents the corresponding backbone network, H, W and C represent the height, width and channel number of the feature map respectively; through the local training stage of the two-stage heterogeneous collaborative perception training framework, the intelligent agent is independently trained to update its single vehicle perception model. 3.The laser-radar-based heterogeneous co-perception 3D target detection method of claim 2, wherein, In S2, the features of the intelligent agents participating in collaboration are aligned through the collaborative training stage, including: Through the collaborative training stage of the heterogeneous collaborative perception training framework, load the models obtained by all collaborative intelligent agents in the first stage training, input the data required for collaborative training to obtain the BEV feature map of each intelligent agent; Then, through a preliminary scale alignment layer, use a 1x1 convolution to preliminarily align the feature maps from different intelligent agents; Through the collaborative training stage of the heterogeneous collaborative perception training framework, the parameter information of all participating collaborative intelligent agents is aligned with the perspective of the ego vehicle; The feature alignment includes: the participating collaborative vehicles communicate the coordinate information to the ego vehicle, and the ego vehicle calculates the coordinate conversion matrix according to the coordinate information of the participating collaborative vehicles to convert the feature maps of the participating collaborative vehicles to the coordinate system of the ego vehicle; The transformed feature representation is denoted as wherein denotes a feature transformation operation.

4. The laser-radar-based heterogeneous co-perception 3D target detection method according to claim 3, characterized in that, In S2, a method of minimizing the differences between the intelligent agents is designed to reduce the differences between the feature maps of the intelligent agents, including: {F'1, F'2,... F'n} for each agent i ,F' j F' k ,...F' N} A multi-scale fusion network is constructed, and the feature maps {F′1, F′2,..., F′ i , F′ j F′ k , ...F′ N} of each agent are input into the multi-scale fusion network; and a feature map representation extracted at each scale is obtained , where α represents the αth scale, i, j, k correspond to agents participating in cooperative perception; Calculate the foreground estimation map S of two intelligent agents participating in collaboration; apply a 1x1 convolution to the feature map of each intelligent agent at each scale, and the foreground estimation map of each intelligent agent at scale a is calculated as follows: Based on the foreground estimation map of each intelligent agent at scale a, calculate the weight matrix between intelligent agents i and j: At each scale a, compute the KL divergence m between the feature maps of agents i, j α : Wherein, N is the total number of collaborative intelligent agents, i, j ∈ {1, 2,..., N}; A is the total number of scales, a ∈ {1, 2,..., A}; calculating the total difference over all scales is: Wherein, M represents the total difference loss; by minimizing aligning feature maps of different agents at each scale, feature fusion is performed for all participating collaborative agents.

5. The laser-radar-based heterogeneous co-perception 3D target detection method according to claim 4, characterized in that, The global view auxiliary reinforcement method is designed to perform domain adversarial training on the global feature map and the single-view feature map through a gradient reversal layer and combining a domain classification head, use global information to enhance the feature representation capability of each intelligent agent, and construct a unified feature space, including: Obtaining feature maps of different agents at scale alpha Input the feature maps of different intelligent agents at scale a into the foreground generator to obtain the foreground estimation map; The foreground estimation map is normalized using a softmax function to generate a weight matrix W for feature fusion; Calculate the weighted feature map of different agents at scale a: the fused feature map at scale a merge with the feature map of the individual view to form a combined feature set The combined feature set Fall is input to the gradient inversion layer, the inverted features are obtained, and the inverted features are input to the domain classifier. It is determined by the domain classifier whether the input features are global fusion features or single-view features. The label of the fusion features is set to 1, the label of the single-view features is set to 0, and the domain classification result is output. ​​ The field classifier network structure is: Two layers of 3*3 convolution layer with padding 1: Y1 = relu(Conv 3×3 (F all )) Y2 = relu(Conv 3×3 (Y1)) Maximum pooling is performed on the input feature map: Y3 = maxpool(Y2) The feature map is flattened: Y3 = flatten(Y2) After transformation through three fully connected layers FC, the original output logits without activation function are obtained; Y5 = dropout(relu(fc(Y4))) Y6 = dropout(relu(fc(Y5))) logits = fc(Y6) Based on the field classification result, the field classification cross-entropy loss at each scale a and the total field classification loss at all scales are calculated.

6. The laser-radar-based heterogeneous co-perception 3D target detection method according to claim 5, characterized in that, The field classification cross-entropy loss at each scale a and the total field classification loss at all scales are calculated, including: The field classification cross-entropy loss at each scale a is calculated: Where y refers to the logits obtained by the domain classifier; The field classification cross-entropy loss at each scale a is calculated:

7. The laser-radar-based heterogeneous co-perception 3D target detection method according to claim 6, characterized in that, The S3 further includes: The total loss function of the training model is calculated: Where β, γ and ∈ are hyperparameters.

8. A laser-radar-based heterogeneous co-perception 3D target detection apparatus for implementing the laser-radar-based heterogeneous co-perception 3D target detection method according to any one of claims 1-7, characterized in that, The device includes: A heterogeneous collaborative perception framework design module is configured to construct a heterogeneous collaborative perception training framework; each vendor type backbone is configured to independently update an agent through the heterogeneous collaborative perception training framework; A consistency enhancement module is configured to perform feature fusion through the heterogeneous collaborative perception training framework after obtaining a BEV feature map of a collaborative vehicle; the consistency enhancement module includes a minimum inter-agent feature difference submodule and a global view auxiliary enhancement submodule; The minimum inter-agent feature difference submodule is configured to design a minimum inter-agent difference method to reduce the difference between the feature maps of the agents. The global view auxiliary enhancement submodule is configured to design a global view auxiliary reinforcement method, perform domain adversarial training on the global feature map and the single-view feature map through a gradient inversion layer and a domain classification head, enhance the feature representation capability of each agent using global information, construct a unified feature space, and complete laser radar-based heterogeneous collaborative perception 3D target detection. 9.A laser radar-based heterogeneous collaborative perception 3D target detection device, comprising: a processor; a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement any one of the laser radar-based heterogeneous collaborative perception 3D target detection methods of claims 1-7. 10.A computer readable storage medium having at least one instruction stored therein, the at least one instruction being loaded and executed by a processor to implement any one of the laser radar-based heterogeneous collaborative perception 3D target detection methods of claims 1-7.

Images