Efficient multi-agent detection method and system based on hybrid cooperative communication framework

Abstract

The application provides an efficient multi-agent detection method and system based on a hybrid cooperative communication framework, comprising: detecting original point cloud information collected by a single agent to obtain a single detection result, a confidence degree and an uncertainty degree thereof; encoding the single detection result based on the confidence degree to obtain a detection regression frame information package; supplementing corresponding original point cloud information based on the uncertainty degree and the single detection result to obtain an original point cloud supplement information package; communicating the detection regression frame information package and the original point cloud supplement information package of each agent among other agents; and fusing the original point cloud information of each agent with the detection regression frame information package and the original point cloud supplement information package of other agents to obtain a perception result. The data mixing mode of the application is beneficial to adaptive compression rate of information and meets various communication requirements; the application uses a standardized information data format, ensures that data is independent of a specific detection model, and improves adaptability across different model configurations.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an efficient multi-agent detection method and system based on a hybrid collaborative communication framework. Background Art

[0002] Multi-agent collaborative perception technology provides a mechanism for sharing complementary perception capabilities within multi-agent systems, thereby promoting an overall improvement in perception capabilities. This technology not only offers a fundamental solution to the inherent challenges of single-agent perception, such as occlusion and long-range sensing, but also expands the application scope of multi-agent systems. Currently, this technology has been widely applied in a variety of fields, including multi-vehicle autonomous driving, multi-robot warehouse automation systems, and multi-drone search and rescue, demonstrating its enormous potential.

[0003] The key challenge facing this emerging field is striking a balance between perception performance and communication bandwidth. Real-world communication systems are often subject to various constraints and struggle to withstand the enormous communication overhead required to transmit complete raw observation data and rich feature information. Therefore, the impact of communication costs cannot be ignored when promoting improved perception performance.

[0004] To optimize collaborative sensing under communication-constrained conditions, previous research has proposed solutions from different perspectives. For example, early collaborative methods focused on transmitting raw observed LiDAR point cloud data, which is suitable for situations with a relatively large communication budget; intermediate collaborative methods reduce the communication load by transmitting hidden features, which is more suitable for scenarios with a moderate communication budget; and later collaborative methods focus on transmitting raw LiDAR point cloud data, which is suitable for situations with a relatively large communication budget. Summary of the Invention

[0005] Long-term collaborative approaches primarily transmit detection results to address highly constrained communication budgets. These strategies can further reduce bandwidth consumption. However, it is important to note that each communication message type has its own applicable communication budget range and has limited adaptability to changing communication budgets.

[0006] In view of the defects in the prior art, the purpose of the present invention is to provide an efficient multi-agent detection method and system based on a hybrid collaborative communication framework.

[0007] According to one aspect of the present invention, an efficient multi-agent detection method based on a hybrid cooperative communication framework is provided, comprising:

[0008] Detect the original point cloud information collected by a single intelligent agent and obtain the single detection result in the form of a box, as well as its confidence and uncertainty;

[0009] Based on the confidence level, encoding the monomer detection result to obtain a detection regression frame information package;

[0010] Based on the uncertainty and the monomer detection result, corresponding original point cloud information is supplemented to obtain an original point cloud supplement information package;

[0011] Communicate the detection regression box information package and the original point cloud supplementary information package of each agent among other agents;

[0012] The original point cloud information of each agent is fused with the detection regression box information package and original point cloud supplementary information package received from other agents to obtain the perception result.

[0013] Preferably, the detection of the original point cloud information collected by a single intelligent agent to obtain a single detection result in the form of a box, and its confidence and uncertainty, includes:

[0014] Generate predicted detection regression boxes using a single 3D detector And use non-maximum suppression NMS to make the i-th agent achieve K sparse detection, specifically:

[0015]

[0016] Each detection regression box is parameterized using confidence and uncertainty as follows:

[0017] o=(x,y,z,l,w,h,θ,c,u x ,u x )

[0018] x,y,z,l,w,h,θ,c,u x ,u y represents the predicted 3D detection center position, length, width, height, yaw angle, detection confidence score, and the uncertainty of the position of the detected object in x and y coordinates, expressed as variance

[0019] Preferably, encoding the monomer detection result based on the confidence level to obtain a detection regression frame information package includes:

[0020]

[0021] in, is the detection regression box information package, Φ select (·) is the selection function, i.e., the regression box packer, is a binary selection matrix indicating whether each of the K detections is selected, where 1 indicates selected and 0 indicates not. The constraint function is:

[0022]

[0023] C i is the confidence score of sparse detection, b (o) The traffic constraint.

[0024] Preferably, the method of supplementing the corresponding original point cloud information based on the uncertainty and the monomer detection result to obtain the original point cloud supplementary information package includes:

[0025]

[0026] is the original point cloud data of agent i, is the detection regression box of agent i, B is the uncertainty, Φ point It is a raw lidar point cloud packager.

[0027] Preferably, the detection result D i , uncertainty U i As a raw lidar point cloud packager Φ point Input, point cloud data As output, the process is as follows:

[0028] First, generate the extended detection box E i , to calculate the uncertainty of the position prediction, that is, the k-th enlarged detection box The parameters come from the kth detection result D i [k]:

[0029] E i =Φ expand (D i ,U i )∈R D

[0030] Then, for each Assign a weight to the observation point cloud in the equation to get the weight matrix W i :

[0031] W i =Φ weight (X i ,E i ,U i ,δ)∈R P

[0032] Among them, the expanded detection frame E i The weight assigned to the points within corresponds to the uncertainty score of the detected target, i.e. u x +u y ,δ means it is not in the detection regression frame The laser point cloud within will be given a default weight value;

[0033] Finally, a weighted sampling function s(·) is used to select the weight W within the communication budget. i to b (x) Point sampling,

[0034]

[0035] Among them, the communication budget is the remaining part after the detection regression box information is transmitted, that is, B is the total communication volume, 8K is the communication volume consumed by the transmitted detection regression box information, 4 is the representation dimension of each point cloud, and b(x) is the number of point clouds that can be transmitted under the communication volume constraint.

[0036] Preferably, the fusing of the original point cloud information of each agent with the received detection regression frame information packages and original point cloud supplementary information packages of other agents to obtain the perception result includes:

[0037] The detection regression frame information package is fused with the detection data of the monomer to generate the detection result:

[0038]

[0039] The detection results are then fused with the detection regression box information package to generate the final collaborative detection results:

[0040]

[0041] Where ∪ represents the union operation, It is the updated detection result after early fusion, that is, the point cloud after the detection result is fused with the update; is the final collaborative detection result after late fusion, that is, merging the regression detection box and applying the non-maximum suppression operation Φ nms (·) The result after eliminating redundant detection.

[0042] Preferably, a hybrid cooperative communication framework is used, and each round uses a detection loss function to supervise object detection. The loss function is:

[0043] L=L reg +L cls +L unc

[0044] Among them, L reg is the smooth L1 loss function for detection box regression, L cls is the cross entropy loss function for object classification, L unc is the Dirac loss function used for uncertainty estimation.

[0045] According to a second aspect of the present invention, there is provided an efficient multi-agent detection system in a hybrid cooperative communication framework, comprising:

[0046] A single detector detects the original point cloud information collected by a single agent and obtains a single detection result in the form of a box, as well as its confidence and uncertainty;

[0047] A detection regression frame packager, which encodes the monomer detection result based on the confidence level to obtain a detection regression frame information package;

[0048] An original laser radar point cloud packager, wherein the original laser radar point cloud packager supplements the corresponding original point cloud information based on the uncertainty and the monomer detection result to obtain an original point cloud supplement information package;

[0049] a communication module, wherein the communication module communicates the detection regression box information package and the original point cloud supplementary information package of each agent between other agents;

[0050] The fusion module fuses the original point cloud information of each intelligent agent with the detection regression frame information package and the original point cloud supplementary information package received from other intelligent agents to obtain a perception result.

[0051] According to the third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it can be used to execute the efficient multi-agent detection method based on a hybrid collaborative communication framework, or to run the efficient multi-agent detection system based on a hybrid collaborative communication framework.

[0052] According to the fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the efficient multi-agent detection method based on the hybrid collaborative communication framework, or to run the efficient multi-agent detection system based on the hybrid collaborative communication framework.

[0053] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0054] The embodiments of the present invention provide an efficient multi-agent detection method and system based on a hybrid collaborative communication framework, which enhances adaptability in three aspects:

[0055] 1. A hybrid transmission of two data types: raw observation point cloud data and detection output data. Detection output data is the most concise representation of the detection target and is the preferred option for the agent to write perception. For instances with accurate perception, this is sufficient. However, for inaccurate detection output results, the raw observation point cloud data is appended to provide more comprehensive perception information. This combination allows for flexible adjustment of the compression ratio to meet overall communication bandwidth requirements.

[0056] 2. Prioritize the most important data in each data type: Each detection output result or raw observation point cloud data is assigned an importance score indicating its perceptual importance, ensuring the best perceptual information relay for each data type.

[0057] 3. Use standardized information data formats, using boxes to represent detection output results and point clouds to represent observation results. This ensures that they are independent of the detection model and improves adaptability across different model configurations. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0059] Figure 1 2 is a framework diagram of an efficient multi-agent detection system based on a hybrid cooperative communication framework in one embodiment of the present invention;

[0060] Figure 2 This is a flowchart of an efficient multi-agent detection method based on a hybrid collaborative communication framework in one embodiment of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several variations and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0062] The technical principle of this invention is that each agent generates a corresponding spatial confidence map, in which each element reflects the perceptual importance of the corresponding spatial region. Based on the spatial confidence map, each agent determines the spatial region in which to communicate. In other words, each agent provides other agents with spatially sparse but perceptually important features through communication. At the same time, each agent can request additional information from other agents in multiple rounds of communication, thereby achieving efficient and mutually beneficial collaboration.

[0063] Based on the above principles, in a preferred embodiment of the present invention, an efficient multi-agent detection method based on a hybrid collaborative communication framework is provided. Figure 1 and Figure 2 As shown, the specific process is as follows:

[0064] S100, detect the original point cloud information collected by a single intelligent agent to obtain the single agent detection result, confidence and uncertainty;

[0065] S200, based on the confidence obtained in S100, encode the monomer detection result obtained in S100 to obtain a detection regression frame information package;

[0066] S300, based on the uncertainty of S100 and the single-unit detection result, supplement the corresponding original point cloud information to obtain the original point cloud supplement information package;

[0067] S400, communicating the detection regression box information package of S200 of each agent and the original point cloud supplementary information package of S300 between other agents;

[0068] S500, the original point cloud information of each intelligent agent is integrated with the detection regression frame information package and the original point cloud supplementary information package of other intelligent agents received after communication in S400 to obtain the perception result.

[0069] The above embodiment proposes a novel hybrid collaboration approach that organically integrates early collaboration (raw observation point cloud data) and later collaboration (detection output data) to achieve efficient and effective collaboration. This approach has two distinct advantages. First, by mixing the two transmission data types of raw observation point cloud data and detection output data, it facilitates adaptive information compression and meets various communication requirements. Second, it uses a standardized information data format, representing detection output results as boxes and observation results as point clouds. This ensures their independence from specific detection models and improves adaptability across different model configurations.

[0070] In a preferred embodiment of the present invention, S100 is implemented to detect the original point cloud information collected by a single intelligent agent to obtain the single agent detection result, confidence level, and uncertainty. The specific process is as follows:

[0071] S101, use a single 3D detector to generate the predicted detection regression box. In order to reduce the repeated boxes, use non-maximum suppression (NMS) to make the i-th agent achieve K sparse detection:

[0072]

[0073] S102, each detection regression box predicted by S101 is parameterized using confidence and uncertainty as

[0074] o=(x,y,z,l,w,h,θ,c,u x ,u x )

[0075] The parameterized detection regression box includes the predicted 3D detection center position, length, width, height, yaw angle, detection confidence score, and the uncertainty of the position of the detected object in x and y coordinates. The uncertainty is expressed as variance.

[0076] In some specific embodiments, the single 3D detector uses PointPillar or CenterPoint.

[0077] In a preferred embodiment of the present invention, S200 is implemented to encode the monomer detection result obtained in S100 based on the confidence obtained in S100 to obtain a detection regression frame information package. The specific process is as follows:

[0078]

[0079] in, is the regression detection box information, Φ select (·) is the selection function, which packages the selected regression box information into regression box detection information, Φ select (·) is a confidence-based detection regression box packer that prioritizes detection regression boxes with higher confidence and provides them with accurately encoded perceptual information; is a binary selection matrix indicating whether each of the K detections is selected, where 1 indicates selected and 0 indicates not. Consider the following proxy constraint problem:

[0080]

[0081] C i is the confidence score of the sparse detection. Even though this optimization problem has hard constraints and non-differentiability of binary variables, it has an analytical solution that naturally satisfies all constraints. This solution is achieved by choosing those i It is obtained by taking the b regression detection boxes with the largest corresponding element ranking.

[0082] In a preferred embodiment of the present invention, S300 supplements the corresponding original point cloud information based on the uncertainty and the single-unit detection result of S100 to obtain an original point cloud supplement information package. The specific process is as follows:

[0083]

[0084] in, is the original point cloud data of agent i, is the detection regression box of agent i, B is the uncertainty, Φ point It is a raw lidar point cloud packer that is uncertainty-based and provides supplementary perception information using points from instances with larger uncertainties.

[0085] Box information is the most concise way to convey the perception target. Naturally, for instances that can be accurately perceived, the box message itself is sufficient, eliminating the need to send redundant raw observation point clouds. On the other hand, for cases where the perception results are less accurate, such as when the points on the object are too sparse or when the object is partially occluded, the box message may be unreliable. In this case, it becomes crucial to transmit supplementary point cloud data, because collaborative detection can improve accuracy compared to individual detection results. Therefore, in a preferred embodiment of the present invention, a point cloud information packer based on uncertainty is used to pack the detection results D i , uncertainty U i As input, it will output point cloud data It is expressed in the following form:

[0086] E i =Φ expand (D i ,U i )∈R D

[0087]

[0088] That is, in the original lidar point cloud packer, first, generate the extended detection box E i , to calculate the uncertainty of the position prediction, that is, the K-th expanded detection box Parameters from D i [k]. Intuitively, uncertainty reflects the possible displacement of the regression box. Enlarging the box can include a reasonable area where the instance may be located.

[0089] Then, for each Assign a weight to the observation point cloud in , so as to obtain the weight matrix W i Located in the extended detection range E i The weight assigned to the points within corresponds to the uncertainty score of the instance, i.e. u x +u y This distribution emphasizes the notion that these points may provide supplementary information for identifying instances with lower accuracy.

[0090] Finally, a weighted sampling function s(·) is used to select b according to the weight within the communication budget. (x) The communication budget is the remaining part after the detection box information is transmitted, that is,

[0091] In the above embodiment, the original point cloud can supplement the inaccurate detection frames caused by insufficient information in the individual detection, obtaining the most complete information, which is then transmitted through communication. The receiving agent can fuse this most complete point cloud information from different agents. The most complete information from multiple agents complements each other, potentially recovering the undetectable parts of the individual.

[0092] In a preferred embodiment of the present invention, in S500, the original point cloud information of each agent is fused with the detection regression frame information package and the original point cloud supplementary information package of other agents received after communication in S400 to obtain a perception result. The specific process is as follows:

[0093] The information package obtained by the original lidar point cloud packager is fused with the detection data of the monomer to generate further detection results. This is the standard early fusion process.

[0094]

[0095] Further detection results are then fused with the information package of the detection regression box packer to generate the final collaborative detection results. This is the standard late fusion process.

[0096]

[0097] Where ∪ represents the union operation, It is the updated detection result after early fusion, that is, the point cloud after the detection result is fused with the update. is the final collaborative detection result after late fusion, that is, merging the regression detection box and applying the non-maximum suppression operation Φ nms (·) The result after eliminating redundant detection.

[0098] In order to better obtain perception and object detection results, in a preferred embodiment of the present invention, a hybrid cooperative communication framework is used, and each round uses a detection loss function to supervise object detection. The loss is:

[0099] L=L reg +L cls +L unc

[0100] Among them, L reg is the smooth L1 loss function for detection box regression, L cls is the cross entropy loss function for object classification, L unc is the Dirac loss function used for uncertainty estimation.

[0101] Based on the same inventive concept, another embodiment of the present invention provides an efficient multi-agent three-dimensional detection system with a hybrid cooperative communication framework, comprising:

[0102] Single-agent detector: Use the single-agent detector to detect the original point cloud information collected by a single agent, and obtain the single-agent detection result in the form of a box, as well as its confidence and uncertainty;

[0103] Detection regression frame packager, using the detection regression frame packager, encodes the individual detection results based on the confidence level to obtain the detection regression frame information package;

[0104] The original lidar point cloud packer uses the original lidar point cloud packer to supplement the corresponding original point cloud information based on uncertainty and single detection results to obtain the original point cloud supplement information package;

[0105] The communication module communicates the detection regression box information package and the original point cloud supplementary information package of each agent between other agents;

[0106] Fusion module,Using the fusion module, the original point cloud information of each,agent is fused with the detection regression box information package and,the original point cloud supplementary information package received from other,agents to obtain the perception result.

[0107] The various steps in the above examples of the present invention can refer to the implementation technology of the corresponding steps of the efficient multi-agent three-dimensional detection method of the hybrid collaborative communication framework in the above embodiment, and will not be repeated here.

[0108] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it can be used to execute the efficient multi-agent three-dimensional detection method of the hybrid collaborative communication framework, or, run the efficient multi-agent three-dimensional detection system of the hybrid collaborative communication framework.

[0109] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the efficient multi-agent three-dimensional detection method of the hybrid collaborative communication framework, or, to execute the efficient multi-agent three-dimensional detection system of the hybrid collaborative communication framework.

[0110] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various modifications or variations within the scope of the claims without affecting the essence of the present invention. The above preferred features may be used in any combination as long as they do not conflict with each other.

Claims

1. An efficient multi-agent detection method based on a hybrid collaborative communication framework, characterized by: include: Detect the original point cloud information collected by a single intelligent agent and obtain the single detection result in the form of a box, as well as its confidence and uncertainty; Based on the confidence level, encoding the monomer detection result to obtain a detection regression frame information package; Based on the uncertainty and the monomer detection result, corresponding original point cloud information is supplemented to obtain an original point cloud supplement information package; Communicate the detection regression box information package and the original point cloud supplementary information package of each agent among other agents; The original point cloud information of each agent is fused with the detection regression box information package and original point cloud supplementary information package received from other agents to obtain the perception result.

2. An efficient multi-agent detection method based on a hybrid collaborative communication framework according to claim 1, characterized in that: The original point cloud information collected by a single intelligent agent is detected to obtain a single detection result in the form of a box, and its confidence and uncertainty, including: Generate predicted detection regression boxes using a single 3D detector And use non-maximum suppression NMS to make the i-th agent achieve K sparse detection, specifically: Each detection regression box is parameterized using confidence and uncertainty as follows: o=(x,y,z,l,w,h,θ,c,u x ,u y ) x,y,z,l,w,h,θ,c,u x ,u y represents the predicted 3D detection center position, length, width, height, yaw angle, detection confidence score, and the uncertainty of the position of the detected object in x and y coordinates, expressed as variance 3. The efficient multi-agent detection method based on a hybrid collaborative communication framework according to claim 2 is characterized in that: The encoding of the monomer detection result based on the confidence level to obtain a detection regression frame information package includes: in, is the detection regression box information package, Φ select (·) is the regression box packer, is a binary selection matrix indicating whether each of the K detections is selected, where 1 indicates selected and 0 indicates not. The constraint function is: C i is the confidence score of sparse detection, b (o) The traffic constraint.

4. The efficient multi-agent detection method based on a hybrid collaborative communication framework according to claim 3 is characterized in that: Based on the uncertainty and the monomer detection results, the corresponding original point cloud information is supplemented to obtain the original point cloud supplementary information package, including: is the original point cloud data of agent i, is the detection regression box of agent i, B is the uncertainty, Φ point It is a raw lidar point cloud packager.

5. The efficient multi-agent detection method based on a hybrid collaborative communication framework according to claim 4 is characterized in that: The test result D i , uncertainty U i As a raw lidar point cloud packager Φ point Input, point cloud data As output, the process is as follows: First, generate the extended detection box E i , to calculate the uncertainty of the position prediction, that is, the k-th enlarged detection box The parameters come from the kth detection result D i [k]: And i =Φ expand (D i ,U i )∈R D Then, for each Assign a weight to the observation point cloud in the equation to get the weight matrix W i : W i =Φ weight (X i ,E i ,U i ,δ)∈R P Among them, the expanded detection frame E i The weight assigned to the points within corresponds to the uncertainty score of the detected target, i.e. u x +u y ,δ means it is not in the detection regression frame The laser point cloud within will be given a default weight value; Finally, a weighted sampling function s(·) is used to select the weight W within the communication budget. i to b (x) Point sampling, Among them, the communication budget is the remaining part after the detection regression box information is transmitted, that is, B is the total communication volume, 8K is the communication volume consumed by the transmitted detection regression box information, 4 is the representation dimension of each point cloud, and b(x) is the number of point clouds that can be transmitted under the communication volume constraint.

6. The efficient multi-agent detection method based on a hybrid collaborative communication framework according to claim 5, characterized in that: The original point cloud information of each agent is fused with the detection regression frame information package and the original point cloud supplementary information package received from other agents to obtain the perception result, including: The detection regression frame information package is fused with the detection data of the monomer to generate the detection result: The detection results are then fused with the detection regression box information package to generate the final collaborative detection results: Where ∪ represents the union operation, It is the updated detection result after early fusion, that is, the point cloud after the detection result is fused with the update; is the final collaborative detection result after late fusion, that is, merging the regression detection box and applying the non-maximum suppression operation Φ nms (·) The result after eliminating redundant detection.

7. The efficient multi-agent detection method based on a hybrid collaborative communication framework according to claim 6, characterized in that: Using the hybrid cooperative communication framework, each round uses the detection loss function to supervise object detection. The loss function is: L=L reg +L cls +L unc Among them, L reg is the smooth L1 loss function for detection box regression, L cls is the cross entropy loss function for object classification, L unc is the Dirac loss function used for uncertainty estimation.

8. An efficient multi-agent detection system based on a hybrid collaborative communication framework, characterized by: include: A single detector detects the original point cloud information collected by a single agent and obtains a single detection result in the form of a box, as well as its confidence and uncertainty; A detection regression frame packager, which encodes the monomer detection result based on the confidence level to obtain a detection regression frame information package; An original laser radar point cloud packager, wherein the original laser radar point cloud packager supplements the corresponding original point cloud information based on the uncertainty and the monomer detection result to obtain an original point cloud supplement information package; a communication module, wherein the communication module communicates the detection regression box information package and the original point cloud supplementary information package of each agent between other agents; The fusion module fuses the original point cloud information of each intelligent agent with the detection regression frame information package and the original point cloud supplementary information package received from other intelligent agents to obtain a perception result.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to perform the method according to any one of claims 1 to 7, or run the system according to claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to perform the method according to any one of claims 1 to 7, or to run the system according to claim 8.

Images