A multi-agent collaborative perception method, device and storage medium

Abstract

The present invention provides a multi-agent collaborative perception method, device and storage medium, which relates to the field of collaborative perception technology. The method includes: obtaining first original point cloud data; obtaining first multi-scale features according to the first original point cloud data; selecting and processing the first multi-scale features to obtain first high-information features; broadcasting the first high-information features to each collaborative agent to obtain second high-information features; performing a second fusion process on the first high-information features and all second high-information features to obtain fused collaborative features; and generating detection result information according to the fused collaborative features. By selecting the obtained high-information value features, the influence of the perception bias caused by noise is reduced, and at the same time, the communication bandwidth consumption is reduced to improve the collaborative efficiency. By using the dual-stage fusion of the first fusion process and the second fusion process, the features with high information value transmitted at different stages are fused to reduce the influence of posture noise in the collaborative perception process.

Description

Technical Field

[0001] The present invention relates to the field of collaborative perception technology, and in particular to a multi-agent collaborative perception method, device and storage medium. Background Art

[0002] An intelligent agent is an entity that can perceive the environment and achieve a certain goal through decision-making and action. With the development of deep learning, intelligent agents have been significantly improved in perception tasks such as semantic segmentation, depth estimation, and target detection and tracking. However, a single intelligent agent is easily affected by occlusion and sparse sensor observation data at a long distance because a single intelligent agent can only perceive the environment from a limited visual perspective.

[0003] Collaborative perception overcomes the visual limitations of a single agent by building a system of multiple agents and sharing perceived information. Collaborative perception takes advantage of multiple viewpoints of the same scene, not only detecting distant and occluded objects, but also improving the detection accuracy in dense areas, enabling a more comprehensive and accurate understanding of the environment, thereby better completing downstream tasks.

[0004] The collaborative perception schemes in the prior art usually assume an ideal perception and communication environment. However, in actual applications, collaborative perception faces some complex scenario challenges, such as multi-source noise interference, perception differences, and other issues. In complex perception scenarios, multi-source noise interference has a great impact on the performance of the collaborative perception system. For example, the posture noise during vehicle positioning, motion blur when detecting fast-moving objects, and perception noise in the original data will lead to inaccurate posture information and other issues. The collaborative perception system may be misled, making the collaborative performance worse than that of a single agent. Therefore, in practical applications, these noises cannot be ignored, and the noise will deteriorate the accuracy and robustness of collaborative perception. In addition, in the existing collaborative perception process, communication between agents consumes a lot of communication bandwidth, and the collaborative efficiency is low.

[0005] Therefore, in the collaborative sensing process, how to overcome the negative impact caused by noise and improve collaborative efficiency is an urgent problem to be solved. Summary of the invention

[0006] The present invention provides a multi-agent collaborative perception method, device and storage medium to solve the problem that in the collaborative perception process of the prior art, the negative impact of noise is not taken into account, resulting in poor perception accuracy and robustness and low collaborative efficiency.

[0007] The present invention provides a multi-agent collaborative perception method, which is applied to a central agent and includes:

[0008] Acquire first original point cloud data;

[0009] Acquire a first multi-scale feature according to the first original point cloud data, where the first multi-scale feature includes features extracted at multiple scales;

[0010] Selecting and processing the first multi-scale features to obtain features whose information value is greater than a preset threshold to form first high-information features;

[0011] Broadcasting the first high-information feature to each collaborative agent to obtain a second high-information feature fed back by each collaborative agent, wherein the second high-information feature is obtained by the collaborative agent performing a first fusion process and a selection process on the first high-information feature and its own second original point cloud data;

[0012] Performing a second fusion process on the first high-information feature and all the second high-information features to obtain a fused collaborative feature;

[0013] According to the fused collaborative features, detection result information is generated.

[0014] According to a multi-agent collaborative perception method provided by the present invention, before generating the detection result information according to the fused collaborative features, the method further includes:

[0015] Obtain historical point cloud data and historical motion features;

[0016] Acquire current motion features according to the historical point cloud data, the historical motion features and the first original point cloud data;

[0017] The fused collaborative feature is processed according to the current motion feature to reduce motion blur.

[0018] According to a multi-agent collaborative perception method provided by the present invention, the current motion feature is obtained according to the historical point cloud data, the historical motion feature and the first original point cloud data, including:

[0019] Convert the historical point cloud data to the current coordinate system and generate a historical bird's-eye view feature map;

[0020] Generate a current bird's-eye view feature map according to the first original point cloud data;

[0021] Generate context information according to the historical bird's-eye view feature map and the current bird's-eye view feature map;

[0022] Performing a first convolution process on the context information to obtain spatial displacement information;

[0023] Performing a second convolution process on the spatial displacement information and the historical bird's-eye view feature map to obtain location point movement features;

[0024] Performing feature channel splicing processing on the position point movement feature and the historical motion feature to form a motion splicing feature;

[0025] Performing channel space attention processing and normalization processing on the motion splicing features to obtain update gate information, where the update gate information represents the probability of retaining the corresponding features at each position;

[0026] The current motion feature is acquired according to the update gate information, the historical motion feature and the position point movement feature.

[0027] According to a multi-agent collaborative perception method provided by the present invention, the first multi-scale feature is selected and processed to obtain a feature whose information value is greater than a preset threshold to form a first high-information feature, including:

[0028] Flattening and connecting the first multi-scale features to form a first feature sequence;

[0029] Performing feature dimension pooling processing on the first feature sequence to generate a semantic confidence sequence, wherein the semantic confidence sequence represents the semantic information value of the feature;

[0030] Processing the semantic confidence sequence according to a preset selection function to form a selection sequence, wherein the selection sequence represents the sequence position of features whose semantic information value is greater than a preset threshold;

[0031] The first high-information feature is generated according to the first feature sequence and the selected sequence.

[0032] According to a multi-agent collaborative perception method provided by the present invention, the first high-information feature includes features at multiple scales, and the second high-information feature includes features at a scale corresponding to the first high-information feature; the first high-information feature is subjected to a second fusion process with all the second high-information features to obtain a fused collaborative feature, including:

[0033] According to the first high-information feature and the second high-information feature, features at the same scale are stacked to obtain a multi-scale aggregate feature;

[0034] Performing maximum pooling processing on the multi-scale aggregation features to obtain maximum pooling multi-scale features, and performing average pooling processing on the multi-scale aggregation features to obtain average pooling multi-scale features;

[0035] Fusing the maximum pooling multi-scale feature with the average pooling multi-scale feature to obtain a multi-scale fusion feature;

[0036] The features at each scale in the multi-scale fusion feature are sampled, converted into features at the same scale and spliced ​​to form the fusion collaborative feature.

[0037] The present invention also provides a multi-agent collaborative perception method, which is applied to collaborative agents and includes:

[0038] Get the central pose data of the central agent;

[0039] Generate a transformation matrix according to the central pose data;

[0040] Acquire second original point cloud data;

[0041] According to the transformation matrix, the second original point cloud data is transformed into the coordinate system of the central intelligent agent to form collaborative perception data;

[0042] Acquire a second multi-scale feature according to the collaborative sensing data;

[0043] Obtain the first high-information feature of the central agent;

[0044] Performing a first fusion process on the first high-information feature and the second multi-scale feature to obtain a multi-scale collaborative feature;

[0045] Selecting and processing the multi-scale collaborative features to obtain features whose information value is greater than a preset threshold to form a second high information value feature;

[0046] The second high-information feature is sent to the central agent, so that the central agent performs a second fusion process on the first high-information feature and the second high-information feature, obtains a fused collaborative feature, and generates detection result information.

[0047] According to a multi-agent collaborative perception method provided by the present invention, the first high-information feature and the second multi-scale feature are subjected to a first fusion process to obtain a multi-scale collaborative feature, including:

[0048] Taking the maximum value of the first high-information feature and the second multi-scale feature for fusion, to obtain a third multi-scale feature;

[0049] Performing pyramid sampling on features of different scales in the third multi-scale features to obtain fourth multi-scale features corresponding to the scales;

[0050] Fusing features corresponding to the same scale among the plurality of fourth multi-scale features to obtain a fifth multi-scale feature;

[0051] Fusing the third multi-scale feature with the fifth multi-scale feature to obtain a sixth multi-scale feature;

[0052] The sixth multi-scale feature is subjected to multi-scale variable self-attention feature fusion processing to obtain the multi-scale collaborative feature.

[0053] According to a multi-agent collaborative perception method provided by the present invention, the step of performing multi-scale variable self-attention feature fusion processing on the sixth multi-scale feature to obtain the multi-scale collaborative feature includes:

[0054] Flattening and connecting the sixth multi-scale features to form a second feature sequence;

[0055] According to the second feature sequence, obtaining a sampling offset sequence, an attention weight sequence, and an attention value sequence;

[0056] The sampling offset sequence, the attention weight sequence and the attention value sequence are fused to obtain the multi-scale collaborative feature.

[0057] The present invention also provides a multi-agent collaborative perception method, which is applied to a collaborative perception system. The collaborative perception system includes multiple agents, including:

[0058] Randomly select one of the intelligent agents from the plurality of intelligent agents as a central intelligent agent, and the remaining intelligent agents as collaborative intelligent agents;

[0059] The central agent executes the above-mentioned multi-agent collaborative perception method applied to the central agent, and the collaborative agent executes the above-mentioned multi-agent collaborative perception method applied to the collaborative agent.

[0060] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a multi-agent collaborative perception method as described above applied to a central agent or a multi-agent collaborative perception method as described above applied to collaborative agents.

[0061] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements a multi-agent collaborative perception method as described above applied to a central agent or a multi-agent collaborative perception method as described above applied to collaborative agents.

[0062] The present invention provides a multi-agent collaborative perception method, device and storage medium, which have at least the following beneficial effects: the central agent extracts features from multiple scales to form a first multi-scale feature based on its own first original point cloud data, and then selects features with information values ​​greater than a preset threshold from the features of multiple scale levels to form a first high-information feature. The central agent shares the first high-information feature with each collaborative agent to obtain a second high-information feature generated by the collaborative agent, and the second high-information feature is generated by performing a first fusion process on the first high-information feature and its own second original point cloud data and also after selection. The central agent performs a second fusion process on the first high-information feature and all the second high-information features to obtain a fused collaborative feature, and then generates detection result information to achieve the effect of collaborative perception. In this way, the first high-information feature and the second high-information feature can reduce the influence of perception bias caused by noise by selecting the high-information value features, while reducing communication bandwidth consumption, which is conducive to improving collaborative efficiency. In addition, the dual-stage fusion of the first fusion process and the second fusion process is used to fuse the features of the high-information value transmitted at different stages to reduce the influence caused by posture noise during collaborative perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0064] Figure 1 It is one of the flow charts of a multi-agent collaborative perception method applied to a central agent provided by the present invention;

[0065] Figure 2 This is the second flow chart of a multi-agent collaborative perception method applied to a central agent provided by the present invention;

[0066] Figure 3 This is the third flow chart of a multi-agent collaborative perception method applied to a central agent provided by the present invention;

[0067] Figure 4 This is a fourth flow chart of a multi-agent collaborative perception method applied to a central agent provided by the present invention;

[0068] Figure 5 This is the fifth flow chart of a multi-agent collaborative perception method applied to a central agent provided by the present invention;

[0069] Figure 6It is one of the flow charts of a multi-agent collaborative perception method applied to collaborative agents provided by the present invention;

[0070] Figure 7 This is the second flow chart of a multi-agent collaborative perception method applied to collaborative agents provided by the present invention;

[0071] Figure 8 This is a third flow chart of a multi-agent collaborative perception method applied to a collaborative agent provided by the present invention;

[0072] Fig. 9 It is a flowchart of a multi-agent collaborative perception method in a training process provided by the present invention;

[0073] Fig.10 It is a schematic diagram of overall data processing in a multi-agent collaborative perception method provided by the present invention;

[0074] Fig.11 It is a schematic diagram of a process of obtaining multi-scale collaborative features in a multi-agent collaborative perception method provided by the present invention;

[0075] Fig.12 It is a schematic diagram of a calculation process for obtaining current motion features in a multi-agent collaborative perception method provided by the present invention;

[0076] Fig.13 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] Combine the following Figure 1-Figure 5 A multi-agent collaborative perception method of the present invention is described, which is applied to a central agent and includes:

[0079] S100: Acquire first original point cloud data;

[0080] S200: Acquire a first multi-scale feature according to the first original point cloud data, where the first multi-scale feature includes features extracted at multiple scales;

[0081] S300: Selecting and processing the first multi-scale features to obtain features with information value greater than a preset threshold to form first high-information features;

[0082] S400: broadcasting the first high-information feature to each collaborative agent to obtain a second high-information feature fed back by each collaborative agent, wherein the second high-information feature is obtained by the collaborative agent performing a first fusion process and a selection process on the first high-information feature and its own second original point cloud data;

[0083] S600: Performing a second fusion process on the first high-information content feature and all the second high-information content features to obtain a fused collaborative feature;

[0084] S800: Generate detection result information according to the fused collaborative features.

[0085] The central agent extracts features from multiple scales based on its own first original point cloud data to form a first multi-scale feature, and then selects features with information values ​​greater than a preset threshold from the features of multiple scale levels to form a first high-information feature. The central agent shares the first high-information feature with each collaborative agent to obtain the second high-information feature generated by the collaborative agent. The second high-information feature is generated by the first high-information feature and its own second original point cloud data after a first fusion process and also after selection. The central agent performs a second fusion process on the first high-information feature and all the second high-information features to obtain a fused collaborative feature, and then generates detection result information to achieve the effect of collaborative perception. In this way, the first high-information feature and the second high-information feature can reduce the impact of perception bias caused by noise by selecting the high-information value features, while reducing communication bandwidth consumption, which is conducive to improving collaborative efficiency. In addition, the dual-stage fusion of the first fusion process and the second fusion process is used to fuse the features with high information values ​​transmitted at different stages to reduce the impact caused by posture noise during collaborative perception.

[0086] In the collaborative perception process, perceptual noise will distort features, which may mislead the collaborative perception results. To address the perceptual bias caused by perceptual noise, features with high information value are selected from features at different scales through feature selection, thereby reducing the impact of perceptual noise. To address the misalignment of collaborative features caused by posture noise, multi-scale feature fusion is used in the two-stage communication between the central agent and the collaborative agent to reduce the negative impact caused by posture noise. In this way, the negative impact caused by noise is reduced in the collaborative perception process.

[0087] The first original point cloud data and the second original point cloud data are data generated by the intelligent agent after detecting objects in the surrounding environment through various sensors, which can reflect the perceived object information in the three-dimensional space, and specifically can include perceived image information, perceived depth information, etc. The first multi-scale feature and the second to sixth multi-scale features and multi-scale collaborative features mentioned later capture information at different levels through features extracted at different scales. The features of each scale contain information of different levels and sizes, which is convenient for reflecting structural information from details to the whole.

[0088] The second highest information content feature is generated by the collaborative agent. The specific process of generating the second highest information content feature is described in detail in a multi-agent collaborative perception method applied to collaborative agents later.

[0089] refer to Figure 2 In some embodiments of the multi-agent collaborative perception method of the present invention, before S800, the method further includes:

[0090] S700: Acquire historical point cloud data and historical motion features;

[0091] S710: Acquire current motion features according to the historical point cloud data, the historical motion features and the first original point cloud data;

[0092] S720: Process the fused collaborative feature according to the current motion feature to reduce motion blur.

[0093] In the collaborative perception process, the movement of objects is affected by motion blur, which affects the accuracy of collaborative perception in target detection. In this regard, by obtaining the recorded historical point cloud data and historical motion features, based on the continuity of motion, the current motion features are obtained from the contextual relationship between the historical point cloud data, historical motion features and the first original point cloud data to compensate for the current fused collaborative features, reduce the impact of motion blur, and help improve the accuracy of the detection result information finally generated based on the fused collaborative features.

[0094] refer to Figure 3 In some embodiments of the multi-agent collaborative perception method of the present invention, the S300 includes:

[0095] S310: Flattening and connecting the first multi-scale features to form a first feature sequence;

[0096] S320: Performing feature dimension pooling processing on the first feature sequence to generate a semantic confidence sequence, where the semantic confidence sequence represents the semantic information value of the feature;

[0097] S330: Processing the semantic confidence sequence according to a preset selection function to form a selection sequence, wherein the selection sequence represents the sequence position of features whose semantic information value is greater than a preset threshold;

[0098] S340: Generate the first high-information feature according to the first feature sequence and the selected sequence.

[0099] The features of multiple scales are flattened based on the position to form a first feature sequence. The sequence position in the first feature sequence corresponds to the image position in the corresponding scale. The first feature sequence is subjected to feature dimension pooling processing to estimate the semantic information value contained in each feature and generate a semantic confidence sequence. According to the preset selection function, the semantic confidence sequence is processed, and a selection sequence is formed according to the sequence position of the features corresponding to the high information value. The first feature sequence is processed according to the selection sequence, and the features with high information value are selected to generate the first high information value feature. In this way, features with low information value can be screened and removed, which is conducive to reducing the amount of data in collaborative perception communication, reducing the consumption of communication bandwidth, and improving collaborative efficiency. At the same time, the perceptual noise has a greater impact on features with low information value, and a smaller impact on features with high information value. Selecting the first high information value feature with less impact is conducive to reducing the impact of perceptual noise.

[0100] In some embodiments of the present invention, the first high information content feature is obtained by selection processing, which can be processed by the following expression:

[0101]

[0102] A=Φ select (m)∈{0,1} 1×L

[0103]

[0104] Among them, m is the semantic confidence sequence; E t is the first characteristic sequence and C is the number of feature channels, L is the length of the first feature sequence; MaxPool(·) and AvgPool(·) represent the maximum pooling and average pooling operations of the feature dimension; Φ select (·) is a selection function used to select the most informative feature; A is a selection sequence, where the position of the sequence where the feature with high information value is located is 1, and the position of the sequence where the feature with low information value is located is 0; B t is the first feature with the highest information content; ⊙ represents the multiplication operation of each element.

[0105] For ease of understanding, here we briefly introduce the first fusion processing process after the collaborative agent obtains the first high-information feature according to the data processing flow:

[0106] refer to Fig.10 and Fig.11 After the collaborative agent obtains the first high-information feature of the central agent, it performs a first fusion process on the first high-information feature and its own second multi-scale feature to obtain a multi-scale collaborative feature, and then selects and processes the multi-scale collaborative feature to form a second high-information feature. The second high-information feature is similar to the first high-information feature and is also obtained after selection. The specific process of selection is basically the same as that of the first high-information feature. After flattening and connecting the multi-scale collaborative features to form a third feature sequence, the processing process of the third feature sequence is the same as that of the first feature sequence. Please refer to the above content and do not repeat it. The second high-information feature also has the effect of reducing the consumption of communication bandwidth, improving collaborative efficiency, and reducing the impact of perceived noise.

[0107] The first high-information feature and the second high-information feature are formed by flattening the first multi-scale feature and the multi-scale collaborative feature and performing a selection process, wherein the sequence position is not changed during the selection process, that is, the correspondence between the sequence position and the image position in the corresponding scale remains unchanged. Therefore, the first high-information feature and the second high-information feature can be converted into a multi-scale form.

[0108] refer to Figure 4 In some embodiments of a multi-agent collaborative perception method of the present invention, the first high-information feature includes features at multiple scales, and the second high-information feature includes features at a scale corresponding to the first high-information feature; S600 includes:

[0109] S610: stacking features at the same scale according to the first high-information feature and the second high-information feature to obtain a multi-scale aggregate feature;

[0110] S620: performing maximum pooling processing on the multi-scale aggregation features to obtain maximum pooling multi-scale features, and performing average pooling processing on the multi-scale aggregation features to obtain average pooling multi-scale features;

[0111] S630: Fusing the maximum pooling multi-scale feature with the average pooling multi-scale feature to obtain a multi-scale fusion feature;

[0112] S640: Sampling features at each scale in the multi-scale fusion feature, converting them into features at the same scale and concatenating them to form the fusion collaborative feature.

[0113] The features of the first high-information feature and the second high-information feature corresponding to the same-scale feature image are stacked to obtain multi-scale aggregate features, and then the same-scale images are further fused after maximum pooling and average pooling to obtain multi-scale fusion features, thereby realizing the fusion of collaborative features. The feature images of different scales in the multi-scale fusion features are then sampled and processed to convert them into images of the same scale, and then spliced ​​to form a fused collaborative feature under a scale feature image. In this way, the multi-scale and high-information features are integrated to facilitate the final target detection and generate detection result information.

[0114] In some embodiments of the present invention, the central agent stacks the feature graphs at all scales included in the first high-information feature and the second high-information feature to form a feature graph of multi-scale aggregated features. Where s is the order of the feature map scale and n is the total number of agents in the collaborative perception scene. Subsequently, the maximum pooling and average pooling operations are applied to And get the feature map of the maximum pooled multi-scale features And the feature map of average pooling multi-scale features Among them, H s and W s Represent the height and width of the feature respectively. The pooled feature maps are concatenated together and further fused through a 3D convolutional layer with a ReLU activation function to obtain multi-scale fused features.

[0115] refer to Figure 5 In some embodiments of the multi-agent collaborative perception method of the present invention, S710 includes:

[0116] S711: converting the historical point cloud data into the current coordinate system and generating a historical bird's-eye view feature map;

[0117] S712: Generate a current bird's-eye view feature map according to the first original point cloud data;

[0118] S713: Generate context information according to the historical bird's-eye view feature map and the current bird's-eye view feature map;

[0119] S714: performing a first convolution process on the context information to obtain spatial displacement information;

[0120] S715: performing a second convolution process on the spatial displacement information and the historical bird's-eye view feature map to obtain a location point movement feature;

[0121] S716: performing feature channel splicing processing on the position point movement feature and the historical motion feature to form a motion splicing feature;

[0122] S717: performing channel space attention processing and normalization processing on the motion stitching features to obtain update gate information, where the update gate information represents the probability of retaining the corresponding features at each position;

[0123] S718: Acquire the current motion feature according to the update gate information, the historical motion feature, and the position point movement feature.

[0124] Taking the coordinate system of the central intelligent agent as the current coordinate system, the historical point cloud data is converted to the current coordinate system and then a historical bird's-eye view feature map is generated. Combined with the current bird's-eye view feature map generated by the current first original point cloud data, context information reflecting feature motion changes can be obtained. The context information is subjected to a first convolution process to obtain spatial displacement information, which can reflect the motion changes caused by the movement of the intelligent agent. A second convolution process is performed in combination with the spatial displacement information and the historical bird's-eye view feature map to obtain the position point movement feature, which can reflect the movement changes of the position point in the image and determine whether the features at each position point are updated. The position point movement feature and the feature channel of the historical motion feature are spliced ​​to form a motion splicing feature, and then the channel space attention process is performed to obtain the overall attention map and then normalize it so that the attention range is between 0 and 1 to form update gate information. The update gate information can reflect the probability of the corresponding feature of each position being retained. The greater the attention corresponding to the position, the greater the probability of the corresponding attention being retained. According to the update gate information, the historical motion feature and the position point movement feature, the current motion feature is obtained and the current motion feature is calculated.

[0125] It can be understood that the current motion feature will be used as the historical motion feature when the next frame is processed.

[0126] refer to Fig.12 , in the figure is the current bird's-eye view feature map, For a historical bird's-eye view of the characteristics, For the historical movement characteristics, The entire calculation process can be completed by the gated motion unit, that is, the current bird's-eye view feature map is input Historical bird's-eye view feature map and historical movement characteristics To the gated motion unit, you can get the current motion features of the output in addition, is the context information, is the spatial displacement information, Move the feature for the location point, To update door information.

[0127] In some embodiments of the present invention, the position point movement feature can be obtained by the following expression:

[0128]

[0129] in, is the moving characteristic of position point p, is the historical bird's-eye view feature map, W is the weight of the convolution, R is the convolution kernel grid, p is the position point, and p n The positions in R are enumerated. Includes the movement characteristics of each location point.

[0130] In some embodiments of the present invention, the update gate information may be obtained by the following expression:

[0131]

[0132] in, To update the door information, Move the feature for the location point, is the historical motion feature, Γ(·) is the channel space attention module, [·,·] represents the feature channel concatenation operation, and σ(·) represents the sigmoid function.

[0133] In some embodiments of the present invention, the current motion feature may be obtained by the following expression:

[0134]

[0135] in, is the current motion feature, To update the door information, For the historical movement characteristics, is the location point moving feature; ⊙ represents the multiplication operation of each element.

[0136] The following will introduce the method and process of the collaborative intelligent agent processing data, which can be cross-referenced with the method and process of the central intelligent agent processing data.

[0137] refer to Figure 6 The present invention also provides a multi-agent collaborative perception method, which is applied to collaborative agents and includes:

[0138] S500: Obtaining the central position data of the central intelligent body;

[0139] S510: Generate a transformation matrix according to the central posture data;

[0140] S520: Acquire second original point cloud data;

[0141] S530: According to the conversion matrix, convert the second original point cloud data into the coordinate system of the central intelligent agent to form collaborative perception data;

[0142] S540: Acquire a second multi-scale feature according to the collaborative sensing data;

[0143] S550: Obtaining the first high-information feature of the central agent;

[0144] S560: Performing a first fusion process on the first high-information feature and the second multi-scale feature to obtain a multi-scale collaborative feature;

[0145] S570: Selecting and processing the multi-scale collaborative features to obtain features with information value greater than a preset threshold to form a second high information value feature;

[0146] S580: Send the second high-information feature to the central agent, so that the central agent performs a second fusion process on the first high-information feature and the second high-information feature, obtains a fused collaborative feature, and generates detection result information.

[0147] The collaborative agent obtains the central pose information of the central agent to generate a transformation matrix to correct the coordinate system of the second original point cloud data generated by itself, and converts it to the coordinate system of the central agent to form collaborative perception data. The subsequent data generated based on the collaborative perception data processing is transmitted back to the central agent, which can keep all data in the central agent corresponding to the same coordinate system for easy processing.

[0148] Extracting the second multi-scale features from the collaborative perception data, and performing the first fusion processing of the first high-information feature obtained from the central intelligent agent and the second multi-scale feature of itself, can be multi-scale enhanced aggregation processing, and the cross-scale layer fusion collection information is used to generate a more robust and comprehensive feature representation, that is, to obtain multi-scale collaborative features, which can effectively resist noise interference, and at the same time, the combination of perception data is conducive to the realization of collaborative perception functions. Selecting and processing the multi-scale collaborative features to obtain the second high-information feature can reduce the impact of perception bias caused by noise, while reducing communication bandwidth consumption, which is conducive to improving collaborative efficiency.

[0149] refer to Figure 7 In some embodiments of the multi-agent collaborative perception method of the present invention, S560 includes:

[0150] S561: Taking the maximum value of the first high-information feature and the second multi-scale feature for fusion to obtain a third multi-scale feature;

[0151] S562: Perform pyramid sampling on features of different scales in the third multi-scale features to obtain fourth multi-scale features corresponding to the scales;

[0152] S563: Fusing features corresponding to the same scale among the plurality of fourth multi-scale features to obtain a fifth multi-scale feature;

[0153] S564: Fusing the third multi-scale feature with the fifth multi-scale feature to obtain a sixth multi-scale feature;

[0154] S565: Perform multi-scale variable self-attention feature fusion processing on the sixth multi-scale feature to obtain the multi-scale collaborative feature.

[0155] refer to Fig.11 , the first high-information feature and the second multi-scale feature are fused based on the maximum value according to the corresponding scale layer, that is, the maximum value is taken from the two to form the third multi-scale feature, and the initial fusion of the features is achieved. The features in different scale layers in the third scale feature are pyramid sampled respectively to obtain the fourth multi-scale feature corresponding to the number of scale layers. The features corresponding to the same scale layer in multiple fourth multi-scale features are fused. Since the fourth multi-scale features are formed by pyramid sampling from different scale layers, the fusion of multiple fourth multi-scale features can realize cross-layer fusion and information collection, which is conducive to generating more robust and comprehensive feature representations, forming the fifth multi-scale feature, and can effectively resist noise interference. Since after multiple rounds of pyramid sampling aggregation, the detail features in the feature map are often lost due to cross-scale layer and cross-semantic fusion, the original third scale feature is fused with the fifth scale feature to compensate for the details and obtain the sixth scale feature. The sixth scale feature is subjected to variable self-attention feature fusion processing to further collect more scale features and reduce the influence of posture noise on feature distortion, forming multi-scale collaborative features, which is conducive to more comprehensive and accurate reflection of features.

[0156] refer to Figure 8 In some embodiments of the multi-agent collaborative perception method of the present invention, the S565 includes:

[0157] S566: Flatten and connect the sixth multi-scale features to form a second feature sequence;

[0158] S567: Acquire a sampling offset sequence, an attention weight sequence, and an attention value sequence according to the second feature sequence;

[0159] S568: Fusing the sampling offset sequence, the attention weight sequence, and the attention value sequence to obtain the multi-scale collaborative feature.

[0160] refer to Fig.11, each scale layer in the sixth multi-scale feature is flattened into one dimension and connected to form a second feature sequence for subsequent processing. The second feature sequence is processed by three functional linear layers, and the sampling offset, attention weight and attention value at the position point are calculated respectively to form a sampling offset sequence, an attention weight sequence and an attention value sequence, and then the three are fused to obtain multi-scale collaborative features, which is conducive to further collecting multi-scale features and reducing the influence of posture noise on feature distortion.

[0161] In some embodiments of the present invention, before the sampling offset sequence, the attention weight sequence and the attention value sequence are fused, the attention weight sequence is subjected to normalized exponential processing.

[0162] In some embodiments of the present invention, the variable self-attention feature fusion processing can be implemented by the following expression processing:

[0163]

[0164] Where MSDA(q) is the output after variable self-attention feature fusion processing at position point q, H is the number of attention heads, S is the number of multi-scale feature maps, and P is the number of sampling points in each scale feature map. h represents the mapping matrix, ψ(·) is the normalized exponential function, and according to the MSDA(q) of each position point q, a multi-scale collaborative feature can be formed

[0165] refer to Fig. 9 In some embodiments of the multi-agent collaborative perception method of the present invention, after the central agent executes S800, the method further includes:

[0166] S900: Obtain status information to determine whether it is a training process;

[0167] S910: When it is a training process, obtain actual label information, and calculate a loss parameter according to the detection result information and the actual label information;

[0168] S920: Update the parameters of the second fusion process according to the loss parameter;

[0169] S930: Send the loss parameter to the collaborative agent so that the collaborative agent updates the parameters of the first fusion process.

[0170] During training, the final detection result information is compared with the actual label information to obtain the detection accuracy, and the loss parameter is calculated. Based on the loss parameter, the parameters of the first fusion processing and the parameters of the second fusion processing are optimized to further improve the detection accuracy of collaborative perception.

[0171] The present invention also provides a multi-agent collaborative perception method, which is applied to a collaborative perception system. The collaborative perception system includes multiple agents, including:

[0172] Randomly select one of the intelligent agents from the plurality of intelligent agents as a central intelligent agent, and the remaining intelligent agents as collaborative intelligent agents;

[0173] The central agent executes the above-mentioned multi-agent collaborative perception method applied to the central agent, and the collaborative agent executes the above-mentioned multi-agent collaborative perception method applied to the collaborative agent.

[0174] Multiple agents form a collaborative perception system, one of which is randomly selected as the central agent, and the other agents are selected as collaborative agents. The central agent and the collaborative agents cooperate with each other to respectively execute the multi-agent collaborative perception method applied to the central agent and the collaborative agent, so that the central agent finally generates detection result information. In this collaborative perception process, by selecting the obtained features with high information value, the influence of perception bias caused by noise can be reduced, and at the same time, the communication bandwidth consumption can be reduced, which is conducive to improving the collaborative efficiency. By using the double-stage fusion of the first fusion processing and the second fusion processing, the features with high information value transmitted at different stages are fused to reduce the influence caused by posture noise in the collaborative perception process, and finally improve the accuracy and reliability of the detection result information.

[0175] It can be understood that an agent, depending on the result of random selection, can act as a central agent or a collaborative agent, and work as either of the two in different collaborative perception processes. Therefore, an agent can simultaneously store and execute relevant program data for a multi-agent collaborative perception method applied to a central agent and a multi-agent collaborative perception method applied to collaborative agents.

[0176] In some embodiments of the collaborative perception system of the present invention, a preset designated intelligent agent may be used as a fixed central intelligent agent. In this case, the central intelligent agent may only store relevant program data for executing a multi-agent collaborative perception method applied to the central intelligent agent, and other collaborative intelligent agents may also only store relevant program data for executing a multi-agent collaborative perception method applied to the collaborative intelligent agent.

[0177] In some embodiments of the present invention, the central intelligent agent shares the detection result information with the collaborative intelligent agent, and the collaborative intelligent agent converts the detection result information into its own coordinate system based on the central posture data of the central intelligent agent and its own posture data to directly obtain the target detection result.

[0178] In some embodiments of the present invention, the intelligent agent may be a sensor module on a car. By forming a collaborative perception system through the sensor modules on the cars within a regional range, the car can more accurately perceive and detect objects in the surrounding environment, which is helpful to avoid vehicles, pedestrians, etc. in blind spots.

[0179] Fig.13 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.13 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the above-mentioned multi-agent collaborative perception method applied to the central agent, or execute the above-mentioned multi-agent collaborative perception method applied to the collaborative agent.

[0180] An electronic device of the present invention can be used as an intelligent agent, and can further be used as a central intelligent agent or a collaborative intelligent agent according to the results of random selection in actual application.

[0181] It can be understood that the memory 830 in the electronic device can simultaneously store logic instructions corresponding to a multi-agent collaborative perception method applied to the central agent and a multi-agent collaborative perception method applied to the collaborative agents.

[0182] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0183] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned multi-agent collaborative perception method applied to the central agent, or to execute the above-mentioned multi-agent collaborative perception method applied to the collaborative agent.

[0184] It can be understood that a non-transitory computer-readable storage medium provided by the present invention can simultaneously store a multi-agent collaborative perception method applied to a central agent and a computer program corresponding to a multi-agent collaborative perception method applied to collaborative agents.

[0185] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0186] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent collaborative perception method, It is characterized in that Applied to the central agent, including: Acquire first original point cloud data; Acquire a first multi-scale feature according to the first original point cloud data, where the first multi-scale feature includes features extracted at multiple scales; Selecting and processing the first multi-scale features to obtain features whose information value is greater than a preset threshold to form first high-information features; Broadcasting the first high-information feature to each collaborative agent to obtain a second high-information feature fed back by each collaborative agent, wherein the second high-information feature is obtained by the collaborative agent performing a first fusion process and a selection process on the first high-information feature and its own second original point cloud data; Performing a second fusion process on the first high-information feature and all the second high-information features to obtain a fused collaborative feature; According to the fused collaborative features, detection result information is generated.

2. A multi-agent collaborative perception method according to claim 1, It is characterized in that Before generating the detection result information according to the fusion collaborative feature, the method further includes: Obtain historical point cloud data and historical motion features; Acquire current motion features according to the historical point cloud data, the historical motion features and the first original point cloud data; The fused collaborative feature is processed according to the current motion feature to reduce motion blur.

3. A multi-agent collaborative perception method according to claim 2, It is characterized in that The acquiring the current motion feature according to the historical point cloud data, the historical motion feature and the first original point cloud data includes: Convert the historical point cloud data to the current coordinate system and generate a historical bird's-eye view feature map; Generate a current bird's-eye view feature map according to the first original point cloud data; Generate context information according to the historical bird's-eye view feature map and the current bird's-eye view feature map; Performing a first convolution process on the context information to obtain spatial displacement information; Performing a second convolution process on the spatial displacement information and the historical bird's-eye view feature map to obtain location point movement features; Performing feature channel splicing processing on the position point movement feature and the historical motion feature to form a motion splicing feature; Performing channel space attention processing and normalization processing on the motion splicing features to obtain update gate information, where the update gate information represents the probability of retaining the corresponding features at each position; The current motion feature is acquired according to the update gate information, the historical motion feature and the position point movement feature.

4. A multi-agent collaborative perception method according to claim 1, It is characterized in that The selecting and processing the first multi-scale features to obtain features with information values ​​greater than a preset threshold to form first high-information features includes: Flattening and connecting the first multi-scale features to form a first feature sequence; Performing feature dimension pooling processing on the first feature sequence to generate a semantic confidence sequence, wherein the semantic confidence sequence represents the semantic information value of the feature; Processing the semantic confidence sequence according to a preset selection function to form a selection sequence, wherein the selection sequence represents the sequence position of features whose semantic information value is greater than a preset threshold; The first high-information feature is generated according to the first feature sequence and the selected sequence.

5. A multi-agent collaborative perception method according to claim 1, It is characterized in that The first high-information feature includes features at multiple scales, and the second high-information feature includes features at a scale corresponding to the first high-information feature; and performing a second fusion process on the first high-information feature and all the second high-information features to obtain a fused collaborative feature includes: According to the first high-information feature and the second high-information feature, features at the same scale are stacked to obtain a multi-scale aggregate feature; Performing maximum pooling processing on the multi-scale aggregation features to obtain maximum pooling multi-scale features, and performing average pooling processing on the multi-scale aggregation features to obtain average pooling multi-scale features; Fusing the maximum pooling multi-scale feature with the average pooling multi-scale feature to obtain a multi-scale fusion feature; The features at each scale in the multi-scale fusion feature are sampled, converted into features at the same scale and spliced ​​to form the fusion collaborative feature.

6. A multi-agent collaborative perception method, It is characterized in that Applications to collaborative agents include: Get the central pose data of the central agent; Generate a transformation matrix according to the central pose data; Acquire second original point cloud data; According to the transformation matrix, the second original point cloud data is transformed into the coordinate system of the central intelligent agent to form collaborative perception data; Acquire a second multi-scale feature according to the collaborative sensing data; Obtain the first high-information feature of the central agent; Performing a first fusion process on the first high-information feature and the second multi-scale feature to obtain a multi-scale collaborative feature; Selecting and processing the multi-scale collaborative features to obtain features whose information value is greater than a preset threshold to form a second high information value feature; The second high-information feature is sent to the central agent, so that the central agent performs a second fusion process on the first high-information feature and the second high-information feature, obtains a fused collaborative feature, and generates detection result information.

7. A multi-agent collaborative perception method according to claim 6, It is characterized in that The performing a first fusion process on the first high-information feature and the second multi-scale feature to obtain a multi-scale collaborative feature includes: Taking the maximum value of the first high-information feature and the second multi-scale feature for fusion, to obtain a third multi-scale feature; Performing pyramid sampling on features of different scales in the third multi-scale features to obtain fourth multi-scale features corresponding to the scales; Fusing features corresponding to the same scale among the plurality of fourth multi-scale features to obtain a fifth multi-scale feature; Fusing the third multi-scale feature with the fifth multi-scale feature to obtain a sixth multi-scale feature; The sixth multi-scale feature is subjected to multi-scale variable self-attention feature fusion processing to obtain the multi-scale collaborative feature.

8. A multi-agent collaborative perception method according to claim 7, It is characterized in that The performing multi-scale variable self-attention feature fusion processing on the sixth multi-scale feature to obtain the multi-scale collaborative feature includes: Flattening and connecting the sixth multi-scale features to form a second feature sequence; According to the second feature sequence, obtaining a sampling offset sequence, an attention weight sequence, and an attention value sequence; The sampling offset sequence, the attention weight sequence and the attention value sequence are fused to obtain the multi-scale collaborative feature.

9. A multi-agent collaborative perception method, It is characterized in that Applied to a collaborative perception system, the collaborative perception system includes multiple agents, including: Randomly select one of the intelligent agents from the plurality of intelligent agents as a central intelligent agent, and the remaining intelligent agents as collaborative intelligent agents; The central agent executes a multi-agent collaborative perception method as described in any one of claims 1 to 5, and the collaborative agent executes a multi-agent collaborative perception method as described in any one of claims 6 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, it implements a multi-agent collaborative perception method as described in any one of claims 1 to 5 or a multi-agent collaborative perception method as described in any one of claims 6 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, it implements a multi-agent collaborative perception method as described in any one of claims 1 to 5 or a multi-agent collaborative perception method as described in any one of claims 6 to 8.

Images