A hybrid collaborative 3D object detection method and system

Abstract

The application relates to the technical field of image processing, and discloses a hybrid collaborative 3D object detection method and system. The method comprises the following steps: a plurality of intelligent agents detect a 3D object to generate intelligent agent poses, object feature maps and object bounding boxes; a plurality of object bounding boxes generated by the plurality of intelligent agents for detecting the same 3D object are aligned to correct the relative poses between the plurality of intelligent agents; a fused object feature map is generated according to the corrected relative poses between the plurality of intelligent agents; and a collaborative object bounding box is generated according to the fused object feature map. The application can effectively suppress the pose noise of collaborative 3D object detection, thereby improving the detection accuracy and detection efficiency of collaborative 3D object detection, and improving the robustness.

Description

Technical Field

[0001] This invention generally relates to the field of image processing technology. Specifically, this invention relates to a hybrid collaborative 3D object detection method and system. Background Technology

[0002] 3D (3D) object detection refers to the detection of parameters such as length, width, height, and distance of three-dimensional objects. In recent years, 3D object detection methods have been widely developed and applied in fields such as autonomous driving, robotics, and virtual worlds. With the rapid development of large-scale datasets and powerful models, 3D object detection methods have also made progress. However, traditional single-agent-based 3D object detection methods have inherent limitations in occluded or distant scenes. Therefore, existing technologies propose utilizing communication between agents (e.g., using V2X communication between vehicles in driving scenarios) to enable multiple agents to share complementary perceptual information, thereby promoting a more comprehensive receptive field.

[0003] Existing technologies have made many improvements in datasets and collaborative methods; however, collaborative 3D object detection methods still have many shortcomings when facing problems such as communication bandwidth limitations, latency, and adversarial attacks.

[0004] Current research on improving collaborative 3D object detection methods focuses on mitigating the negative impact of pose errors. This is because, in order to share effective information, multiple agents need precise poses to synchronize their individual data in a consistent spatial coordinate system. However, the 6DoF (Degrees of Freedom) poses estimated by each agent's localization module are not perfect in reality and often contain pose noise, which fundamentally reduces the quality of collaboration.

[0005] To address pose noise in collaborative 3D object detection methods, existing techniques propose learning pose correction through pose regression modules; or using semantic labels of 3D points to find the correspondence in the point cloud and thereby correct the relative pose. However, these existing techniques require the real pose in the training data, or although pose noise in the training data can be corrected offline, the labeling process may be both expensive and imperfect. Summary of the Invention

[0006] To at least partially address the aforementioned problems in the prior art, this invention proposes a hybrid collaborative 3D object detection method, comprising the following steps:

[0007] Multiple agents detect 3D objects to generate agent poses, object feature maps, and object bounding boxes;

[0008] Align the bounding boxes of multiple objects generated by multiple agents detecting the same 3D object to correct the relative pose between the multiple agents;

[0009] Generate a fused object feature map based on the corrected relative poses between multiple agents; and

[0010] Generate the bounding box of the collaborative object based on the feature map of the fused object.

[0011] In one embodiment of the present invention, the detection of a 3D object by multiple intelligent agents is expressed as follows:

[0012] F i B i =f detection (O i ),

[0013] Among them O i f represents the observation input of the i-th agent to the 3D object. detection This indicates the action performed by the detection module of the intelligent agent, F i Represents the object feature map and B generated by the i-th agent for the 3D object. i This represents the bounding box generated by the i-th agent for the 3D object.

[0014] In one embodiment of the present invention, the hybrid collaborative 3D object detection method further includes estimating the uncertainty of the bounding box, including the following steps:

[0015] The boundary box B i Parameterization, expressed as follows:

[0016]

[0017] Where b represents the parameterized bounding box, Indicates the 3D center position of the bounding box. Indicates the length of the bounding box, Indicates the width of the bounding box, Indicates the height of the bounding box, Indicates the yaw angle of the bounding box, The variance of the center position of the bounding box, and This represents the yaw angle variance of the bounding box;

[0018] Model the center positions x and y and calculate the loss L with respect to the center positions x and y. x and L y ;

[0019] yaw angle Model and calculate the yaw angle. loss L θ ;as well as

[0020] The summation loss function is the total loss L for training the agent to detect and overcome uncertainties. total It can be expressed as the following formula:

[0021] L total =L cls +α reg L reg +α center (L x +L y )+α θ L θ ,

[0022] Where L cls L represents the cross-entropy loss for target classification. reg It is the loss from smoothing L1 box parametric regression, α reg α center α θ This represents the hyperparameters used for balancing.

[0023] In one embodiment of the present invention, aligning multiple object bounding boxes generated by multiple agents detecting the same 3D object to correct the relative pose of the agents includes generating an agent-object pose graph and optimizing the agent-object pose graph, expressed as the following formula:

[0024] {ξ′ j→i} j =f correction ({B j ,ξ j} j＝1,2,…,N ),

[0025] Where ξ′ j→i f represents the corrected relative pose from the perspective of the i-th agent to that of the j-th agent. correction This represents the action performed by the pose correction module of the intelligent agent, ξ j Represents the pose of the j-th agent and B j This represents the bounding box generated by the i-th agent for the 3D object.

[0026] In one embodiment of the present invention, generating an agent-object pose graph and optimizing the agent-object pose graph includes the following steps:

[0027] Generate intelligent agent - object pose graph in Represents the set of intelligent agent nodes, Let ε represent the set of object nodes and let ε represent the set of edges. The set of agent nodes includes agent pose nodes, and the set of object nodes includes object pose nodes. The set of edges ε represents the detection relationship between the agent and the object. When the agent detects a bounding box on the object, the corresponding agent pose node and the object pose node are connected to form an edge.

[0028] The optimization problem of agent-object pose graph can be expressed as follows:

[0029]

[0030] Where χ k Represents the pose and z-axis of the k-th object. jk This represents the relative pose of the k-th object from the perspective of the j-th agent, e. jk Represents the pose consistency error vector. Represents the motion composition operator, ξ -1 Indicates reversed pose, Ω jk This represents the confidence information matrix, wherein the diagonal elements of the information matrix are determined based on the estimation of the uncertainty of the bounding box; and

[0031] The optimization problem of the agent-object pose graph is iteratively optimized to generate the corrected relative pose from the angle of the i-th agent to the angle of the j-th agent, expressed as follows:

[0032]

[0033] In one embodiment of the present invention, generating a fused object feature map based on the corrected relative poses among multiple agents and generating a cooperative object bounding box based on the fused object feature map includes the following steps:

[0034] The i-th agent determines the relative pose ξ′ based on the corrected position. j→i The object feature map F of the j-th agent. j Transformed into the object feature map F of the i-th agent i Transformation characteristics M with the same coordinate system j→i It can be expressed as the following formula:

[0035] M j→i =f transform (F j ,ξ′ j→i ),

[0036] Where f transform This represents the action performed by the agent's transformation module;

[0037] The generated feature map of the fused object is represented by the following formula:

[0038] F′i =f fusion ({M j→i} j＝1,2,…,N ),

[0039] F ′ i f represents the feature map of the fused object after the i-th agent incorporates features from other agents. fusion This represents the actions performed by the fusion module of the intelligent agent; and

[0040] The bounding boxes of the collaborative objects are generated based on the feature maps of the fused objects, expressed as follows:

[0041] B' i =f decoder (F' i ),

[0042] Among them B ′ i Represents the bounding box of the collaborative object, f decoder This indicates the action performed by the decoding output module of the intelligent agent.

[0043] In one embodiment of the present invention, generating a fused object feature map through multi-scale fusion includes the following steps:

[0044]

[0045] in Represents the cooperative characteristics at the l-th spatial scale, g l Let fuse represent the residual layer with a downsampling coefficient of 2, and u represent the attention operation along the agent dimension. L represents the transpose convolution upsampling operator relative to the l-th scale, and concat represents the concatenation operator along the feature channel dimension.

[0046] This invention also proposes a hybrid collaborative 3D object detection system, comprising multiple agents configured to detect 3D objects and generate collaborative object bounding boxes. The agents include:

[0047] The detection module is configured to detect 3D objects to generate smart agent poses, object feature maps, and object bounding boxes;

[0048] The pose correction module is configured to align multiple object bounding boxes generated by multiple agents detecting the same 3D object in order to correct the relative pose of the agents.

[0049] The transformation module is configured to transform the object feature maps of multiple agents to the same coordinate system based on the corrected relative poses between the agents.

[0050] The fusion module is configured to generate a fused object feature map based on the corrected relative poses between multiple agents; and

[0051] The decoding output module is configured to generate a collaborative object bounding box based on the fused object feature map.

[0052] The present invention also proposes a computer system comprising:

[0053] A processor, configured to execute machine-executable instructions; and

[0054] A memory storing machine-executable instructions that, when executed by a processor, perform steps according to the hybrid cooperative 3D object detection method.

[0055] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps according to the hybrid cooperative 3D object detection method.

[0056] The hybrid cooperative 3D object detection method derived from the technical solution of this invention can achieve the following technical effects in the field of image processing technology: effectively suppressing pose noise in cooperative 3D object detection, thereby improving the detection accuracy and efficiency, and enhancing robustness. Extensive experiments on simulated and real datasets (including OPV2V, V2X-Sim 2.0, and DAIR-V2X) for LiDAR-based 3D object detection tasks revealed that this method achieves a 75% reduction in relative pose error (measured at the median), and achieves at least a 12% performance improvement compared to other methods in cooperative 3D object detection tasks with pose errors. Its principle is to use the estimation uncertainty of the bounding box as a key scene landmark for subsequent relative pose correction; to improve pose consistency by optimizing the relative pose relationship between the agent and the detected objects in the scene through agent-object pose graph optimization; and to further mitigate the impact of pose error by comprehensively aggregating cooperative information from multiple spatial scales through a multi-scale intermediate fusion method. Attached Figure Description

[0057] To further illustrate the advantages and other features of the various embodiments of the present invention, a more specific description of the embodiments of the present invention will be presented with reference to the accompanying drawings. It is understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by the same or similar reference numerals for clarity.

[0058] Figure 1 A computer system implementing the system and / or method according to the present invention is shown.

[0059] Figure 2 The diagram illustrates a flowchart of a hybrid collaborative 3D object detection method according to one embodiment of the present invention.

[0060] Figure 3 The diagram illustrates a framework schematic of a hybrid collaborative 3D object detection method according to one embodiment of the present invention. Detailed Implementation

[0061] It should be noted that the components in the various figures may be shown exaggeratedly for illustrative purposes and are not necessarily to scale. In each figure, the same reference numerals are used for components that are identical or have the same function.

[0062] In this invention, unless otherwise specified, "arranged on," "arranged above," and "arranged on" do not exclude the possibility of an intermediate element between them. Furthermore, "arranged on or above" merely indicates the relative positional relationship between two components, and in certain cases, such as when the product orientation is reversed, it can also be converted to "arranged below or under," and vice versa.

[0063] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0064] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0065] It should also be noted that, in the embodiments of the present invention, only a portion of the components or parts may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, necessary components or parts can be added as needed for specific scenarios. Furthermore, unless otherwise stated, features in different embodiments of the present invention can be combined with each other. For example, a feature in the second embodiment can replace a corresponding or functionally identical or similar feature in the first embodiment, and the resulting embodiment will also fall within the scope of disclosure or description of this application.

[0066] It should also be noted that, within the scope of this invention, the terms "same," "equal," and "equal to" do not imply that the two values ​​are absolutely equal, but rather allow for a certain reasonable margin of error. In other words, the terms also encompass "substantially the same," "substantially equal," and "substantially equal to." Similarly, in this invention, the directional terms "perpendicular to," "parallel to," etc., also encompass the meanings of "substantially perpendicular to" and "substantially parallel to."

[0067] Furthermore, the numbering of the steps in the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps may be executed in different orders.

[0068] In this invention, the modules of the system according to the invention can be implemented using software, hardware, firmware, or a combination thereof. When a module is implemented using software, its function can be implemented through computer program flow. For example, the module can be implemented using code segments (such as code segments in languages ​​like C and C++) stored in a storage device (such as a hard disk, memory, etc.), wherein the corresponding function of the module can be implemented when the code segment is executed by a processor. When a module is implemented using hardware, its function can be implemented by setting a corresponding hardware structure. For example, the module's function can be implemented by hardware programming a programmable device such as a field-programmable gate array (FPGA), or by designing an application-specific integrated circuit (ASIC) that includes multiple transistors, resistors, capacitors, and other electronic devices. When a module is implemented using firmware, the module's function can be written into a read-only memory such as an EPROM or EEPROM in the form of program code, and the corresponding function of the module can be implemented when the program code is executed by a processor. In addition, some functions of the module may need to be implemented by separate hardware or by working in cooperation with the hardware. For example, the detection function is implemented by the corresponding sensor (such as a proximity sensor, accelerometer, gyroscope, etc.), the signal transmission function is implemented by the corresponding communication device (such as a Bluetooth device, infrared communication device, baseband communication device, Wi-Fi communication device, etc.), the output function is implemented by the corresponding output device (such as a display, speaker, etc.), and so on.

[0069] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0070] Figure 1 A computer system 100 implementing the system and / or method according to the present invention is shown. Unless otherwise specified, the method and / or system according to the present invention can be implemented in... Figure 1 The invention may be implemented in the computer system 100 shown to achieve the objectives of the invention, or the invention may be implemented in a distributed manner in multiple computer systems 100 according to the invention via a network, such as a local area network or the Internet. The computer system 100 of the invention may include various types of computer systems, such as handheld devices, laptop computers, personal digital assistants (PDAs), multiprocessor systems, microprocessor-based or programmable consumer electronic devices, network PCs, minicomputers, mainframes, network servers, tablet computers, etc.

[0071] like Figure 1As shown, the computer system 100 includes a processor 111, a system bus 101, a system memory 102, a video adapter 105, an audio adapter 107, a hard disk drive interface 109, an optical drive interface 113, a network interface 114, and a Universal Serial Bus (USB) interface 112. The system bus 101 can be any of several bus architecture types, such as a memory bus or memory controller, a peripheral bus, and a local bus using various bus architectures. The system bus 101 is used for communication between the various bus devices. In addition to... Figure 1 Other bus devices or interfaces besides those shown are also conceivable. System memory 102 includes read-only memory (ROM) 103 and random access memory (RAM) 104, wherein ROM 103 may, for example, store basic input / output system (BIOS) data for basic routines used to implement information transfer at startup, while RAM 104 provides the system with faster access speed running memory. Computer system 100 also includes a hard disk drive interface 109 for reading and writing to hard disk 110, an optical drive interface 113 for reading and writing to optical media such as CD-ROM, etc. Hard disk 110 may, for example, store operating system and application programs. Drives and their associated computer-readable media provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data for computer system 100. Computer system 100 may also include a video adapter 105 for image processing and / or image output, which is used to connect output devices such as display 106. The computer system 100 may also include an audio adapter 107 for audio processing and / or audio output, which is used to connect output devices such as speakers 108. Furthermore, the computer system 100 may also include a network interface 114 for network connectivity, wherein the network interface 114 can be connected to the Internet 116 via a network device such as a router 115, wherein the connection can be wired or wireless. Additionally, the computer system 100 may also include a Universal Serial Bus (USB) interface 112 for connecting peripheral devices, wherein the peripheral devices include, for example, a keyboard 117, a mouse 118, and other peripheral devices such as microphones, cameras, etc.

[0072] When the present invention is Figure 1 When implemented on the computer system 100, it can effectively suppress pose noise in cooperative 3D object detection, thereby improving the detection accuracy and efficiency of cooperative 3D object detection, and also enhancing robustness.

[0073] Furthermore, the embodiments can be provided as computer program products that may include one or more machine-readable media on which machine-executable instructions are stored, which, when executed by one or more machines such as a computer, computer network, or other electronic equipment, may cause one or more machines to perform operations according to the embodiments of the present invention. Machine-readable media may include, but are not limited to, floppy disks, optical disks, CD-ROMs (compact disc read-only memory) and magneto-optical disks, ROMs (read-only memory), RAMs (random access memory), EPROMs (erasable programmable read-only memory), EEPROMs (electrically erasable programmable read-only memory), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0074] Furthermore, various embodiments can be downloaded as computer program products, wherein the program can be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) using one or more data signals implemented and / or modulated by a carrier wave or other propagation medium. Therefore, the machine-readable medium used herein may include such a carrier wave, but this is not required.

[0075] Figure 2 A flowchart illustrating a hybrid collaborative 3D object detection method according to one embodiment of the present invention is shown. Figure 2 As shown, the method may include the following steps:

[0076] Step 201: Multiple agents detect 3D objects to generate agent poses, object feature maps, and object bounding boxes.

[0077] Step 202: Align the bounding boxes of multiple objects generated by multiple agents detecting the same 3D object to correct the relative poses between the multiple agents.

[0078] Step 203: Generate a fusion object feature map based on the corrected relative poses between multiple agents.

[0079] Step 204: Generate the bounding box of the collaborative object based on the feature map of the fused object.

[0080] Figure 3 This diagram illustrates a framework schematic of a hybrid collaborative 3D object detection method according to one embodiment of the present invention. The following is a detailed explanation in conjunction with... Figure 3 Detailed descriptions of each embodiment Figure 2 The methods and steps in the text.

[0081] In step 201, single-agent 3D object detection can be expressed as follows:

[0082] F i Bi =f detection (O i ),

[0083] Among them O i f represents the observation input of the i-th agent. detection This represents the processing performed by the agent's 3D object detector, F i B represents the intermediate features generated by the i-th agent. i Let represent the bounding box estimated by the i-th agent. The agent's 3D object detector can be an existing PointPillar.

[0084] Furthermore, the bounding box B can be calculated. i The estimation uncertainty is used as a key scene landmark for subsequent relative pose correction, because incorrect detection may lead to incorrect relative pose correction, and the bounding box B of each agent... i The estimated uncertainty can provide a useful confidence level to exclude suboptimal detections.

[0085] Calculate the bounding box B i The estimation of uncertainty may include the following steps:

[0086] The boundary box B i Parameterization, expressed as follows:

[0087]

[0088] Where b represents the parameterized bounding box, Indicates the 3D center position of the bounding box. length, Indicates the width of the bounding box, Indicates the height of the bounding box, Indicates the yaw angle of the bounding box, The variance of the center position of the bounding box, and This represents the variance of the yaw angle of the bounding box;

[0089] Model the center positions x and y and calculate the loss L with respect to the center positions x and y. x and L y ;

[0090] yaw angle Model and calculate the yaw angle. loss L θ ;as well as

[0091] The summation loss function is the total loss L for training the agent's detection and uncertainty estimation. total It can be expressed as the following formula:

[0092] L total =L cls +α reg L reg +α center (L x +L y )+α θ L θ ,

[0093] Where L cls L represents the cross-entropy loss for target classification. reg It is the loss from smoothing L1 box parametric regression, α reg α center α θ This represents the hyperparameters used for balancing.

[0094] The loss L at the center position x and y is calculated. x and L y In the process of calculating L x For example, the center position x can be modeled as a Gaussian distribution. And minimize the Gaussian distribution The KL divergence between the true labeled delta distribution P(x) = δ(x - x0) is used to calculate the loss L with respect to the center location x. x It can be expressed as the following formula:

[0095]

[0096] Where x0 represents the coordinates of the actual center position.

[0097] At yaw angle When modeling, the yaw angle can be modeled as a von-Mises distribution based on its periodicity, and the von-Mises distribution can be minimized. The KL divergence between the true labeled delta distribution function P(θ) = δ(θ - θ0) is used to calculate the loss L with respect to the yaw angle θ. θ It can be expressed as the following formula:

[0098]

[0099] Where θ0 represents the true yaw angle, I0(·) denotes the 0th-order modified Bessel function;

[0100] When s is large, another regularization term λ can be added to aid gradient descent optimization. elu ELU(sc), where ELU(·) represents the exponential linear unit (ELU), and c represents the hyperparameter controlling the effect of ELU.

[0101] In step 202, the relative pose can be corrected based on all noisy poses obtained from other agents and the detected 3D target bounding boxes, as expressed in the following equation:

[0102] {ξ′ j→i} j =f correction ({B j ,ξ j} j＝1,2,…,N ),

[0103] Where ξ′ j→i Let ξi = (xi, yi, zi, θi, φi, ψi) be the relative pose from the perspective of the i-th agent to the j-th agent, after correction by the agent-object pose graph. θi is the 6DoF pose of the i-th agent. i ,φ i ,ψ i These are yaw angle, pitch angle, and roll angle.

[0104] After single-agent detection, the i-th agent shares three types of information, including the pose ξ estimated by the agent's own localization module. i The bounding box detected by the i-th agent and the feature map F generated by the i-th agent. i In step 202, the pose can be corrected using the first two types of information, feature fusion is performed using the third type of message, and since box detection only has the yaw angle for rotation measurement, the agent's pose is simplified as follows: ξ i =(x i ,y i ,θ i ).

[0105] In hybrid collaborative 3D object detection, each agent needs to correct its relative pose in order to reliably fuse feature maps from other agents. To achieve this, the core is to align multiple bounding boxes of the same target detected by multiple agents. Therefore, an agent-object pose graph can be generated to represent the relationship between the agent and the detected object, and this agent-object pose graph can be optimized to achieve pose alignment.

[0106] Specifically, the i-th agent can receive messages from all other agents and construct its internal agent-object pose graph. The agent-object pose graph is a bipartite graph, in which relationships between agents are modeled and objects are detected. The node set in the agent-object pose graph... It consists of all N intelligent agents, a node set. The model includes all objects within the scene, obtained through spatial clustering of similar bounding boxes of 3D objects received by all agents. The edge set ε reflects the detection relationships between agents and 3D objects, where an agent connects to a 3D object as an edge when it detects a bounding box on the object. The agent-object pose graph is constructed locally for each agent but is identical for all agents. Furthermore, in the agent-object pose graph, each node is associated with a pose.

[0107] For the j-th agent in the scene, its pose ξ j Estimated by its localization module; for the k-th object in the scene, its pose χ k It is sampled from multiple bounding boxes within the same box family, essentially representing a unique object detected by multiple agents. Accordingly, each edge in ε reflects the relative pose between the agent and the object.

[0108] by Let z be the relative pose of the k-th object from the perspective of the j-th agent, where z jk It is naturally obtained through the detection output of the j-th agent.

[0109] Ideally, the pose of a target should be consistent from the perspective of multiple agents; that is, the pose consistency error vector... It should be zero, where ° is the motion composition operator, equivalent to multiplying their corresponding homogeneous transformation matrices. ξ can be... -1 This is represented as the inverse pose, which is equivalent to reversing the corresponding transformation matrix.

[0110] To promote this pose consistency, the following optimization problem can be considered:

[0111]

[0112] in Similar to the information matrix in graph-based SLAM, this matrix measures confidence in the measurement, with its diagonal elements derived from uncertainty estimates of the bounding box. The optimization described above shares a similar formula with graph-based SLAM (simultaneous localization and mapping), but the semantics differ: graph-based SLAM aligns the same object across multiple time stamps, whereas we align the same target at the same time stamp, but it is detected by multiple agents. This typical graph optimization can be solved using Gaussian-Newton or Levenberg-Marquardt algorithms.

[0113] During the optimization iteration, the pose of the ego agent can be fixed while the poses of all other agents and objects are updated. This is because a good initial guess {ξ} has already been given. j ,χ k The agent-object pose graph typically converges well and quickly. Finally, the relative pose is calculated.

[0114] In steps 203 and 204, a multi-scale data fusion strategy can be used to aggregate intermediate features from multiple spatial resolutions.

[0115] Through the corrected relative pose ξ′ j→i The i-th agent can use the features F of the j-th agent. j Perform spatial transformations and obtain its self-characteristic F i Features M with the same coordinate system j→i Furthermore, robust multi-scale fusion is used to update features, where M... i→i =F i Finally, the fused features are used to obtain the final detection:

[0116] M j→i =f transform (F j ,ξ′ j→i ),

[0117] F' i =f fusion ({M j→i} j＝1,2,…,N ),

[0118] B' i =f decoder (F' i ),

[0119] Where F ′ i B is the aggregated feature of the i-th agent after fusing features from other agents. ′ i It is the 3D target detection bounding box output after collaboration.

[0120] After spatial alignment, each agent aggregates collaborative information from other agents and obtains more information features. However, even after relative pose correction, misalignment between feature maps may still exist. To further mitigate the impact of pose noise, a multi-scale fusion method can be employed, fusing features from multiple spatial scales. Finer-scale features can provide more detailed geometric and semantic information, while coarser-scale features are less sensitive to pose errors. A multi-scale structure can combine the advantages of both and produce information-rich and robust features. Mathematically, let... This is the collaborative feature at the l-th spatial scale. Then multi-scale fusion becomes...

[0121]

[0122]

[0123] Where g l It is the l-th residual layer with a downsampling coefficient of 2. fuse is an attention operation along the agent dimension. L `f` is the transpose convolution upsampling operator relative to the `l`-th scale, and `concat` is the concatenation operator along the feature channel dimension. The fused features F′ i It will be decoded into the final detected 3D target bounding box B′ i .

[0124] In one embodiment of the present invention, the above method can be implemented by a hybrid collaborative 3D object detection system, the hybrid collaborative 3D object detection system comprising multiple agents, wherein the multiple agents are configured to detect 3D objects and generate collaborative object bounding boxes, the agents comprising:

[0125] The detection module is configured to detect 3D objects to generate agent pose, object feature map, and object bounding box;

[0126] The pose correction module is configured to align multiple object bounding boxes generated by multiple agents detecting the same 3D object in order to correct the relative pose of the agents.

[0127] The transformation module is configured to transform the object feature maps of multiple agents to the same coordinate system based on the corrected relative poses between the agents.

[0128] The fusion module is configured to generate a fused object feature map based on the corrected relative poses between multiple agents; and

[0129] The decoding output module is configured to generate a collaborative object bounding box based on the fused object feature map.

[0130] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A hybrid collaborative 3D object detection method, characterized in that, Includes the following steps: Multiple agents detect 3D objects to generate agent poses, object feature maps, and object bounding boxes; Align the bounding boxes of multiple objects generated by multiple agents detecting the same 3D object to correct the relative pose between the multiple agents; Generate a fusion object feature map based on the corrected relative poses between multiple agents; as well as Generate bounding boxes for collaborative objects based on the feature map of the fused objects; The method further includes estimating the uncertainty of the bounding box, including the following steps: The boundary box B i Parameterization, expressed as follows: Where b represents the parameterized bounding box, Indicates the 3D center position of the bounding box. Indicates the length of the bounding box, Indicates the width of the bounding box, Indicates the height of the bounding box, Indicates the yaw angle of the bounding box, The variance of the center position of the bounding box, and This represents the yaw angle variance of the bounding box; Model the center positions x and y and calculate the loss L with respect to the center positions x and y. x and L y ; yaw angle Model and calculate the yaw angle. loss L θ ;as well as The summation loss function is the total loss L for training the agent to detect and overcome uncertainties. total It can be expressed as the following formula: L total =L cls +a reg L reg +a center (L x +L y )+a θ L θ , Where L cls L represents the cross-entropy loss for target classification. reg It is the loss from smoothing L1 box parametric regression, α reg α center α θ This represents the hyperparameters used for balancing.

2. The hybrid collaborative 3D object detection method according to claim 1, characterized in that, The detection of 3D objects by multiple agents can be represented by the following formula: F i ,B i =f detection (O i ), Among them O i f represents the observation input of the i-th agent to the 3D object. detection This indicates the action performed by the detection module of the intelligent agent, F i Represents the object feature map and B generated by the i-th agent for the 3D object. i This represents the bounding box generated by the i-th agent for the 3D object.

3. The hybrid collaborative 3D object detection method according to claim 2, characterized in that, Aligning multiple bounding boxes generated by multiple agents detecting the same 3D object to correct the relative pose of the agents includes generating an agent-object pose graph and optimizing the agent-object pose graph, expressed as the following formula: {ξ′ j→i } j =f correction ({B j ,x j } j＝1,2,…,N ), Where ξ′ j→i f represents the corrected relative pose from the perspective of the i-th agent to that of the j-th agent. correction This represents the action performed by the pose correction module of the intelligent agent, ξ j Represents the pose of the j-th agent and B j This represents the bounding box generated by the j-th agent for the 3D object.

4. The hybrid collaborative 3D object detection method according to claim 3, characterized in that, Generating and optimizing the agent-object pose graph includes the following steps: Generate intelligent agent - object pose graph in Represents the set of intelligent agent nodes, Let ε represent the set of object nodes and let ε represent the set of edges. The set of agent nodes includes agent pose nodes, and the set of object nodes includes object pose nodes. The set of edges ε represents the detection relationship between the agent and the object. When the agent detects a bounding box on the object, the corresponding agent pose node and the object pose node are connected to form an edge. The optimization problem of agent-object pose graph can be expressed as follows: Where χ k Represents the pose and z-axis of the k-th object. jk This represents the relative pose of the k-th object from the perspective of the j-th agent, e. jk Represents the pose consistency error vector. Represents the motion composition operator, ξ -1 Indicates reversed pose, Ω jk This represents the confidence information matrix, wherein the diagonal elements of the information matrix are determined based on the estimation results of the uncertainty of the bounding box; as well as The optimization problem of the agent-object pose graph is iteratively optimized to generate the corrected relative pose from the angle of the i-th agent to the angle of the j-th agent, expressed as follows:

5. The hybrid collaborative 3D object detection method according to claim 4, characterized in that, Generating a fused object feature map based on the corrected relative poses among multiple agents and generating a collaborative object bounding box based on the fused object feature map includes the following steps: The i-th agent determines the relative pose ξ′ based on the corrected position. j→i The object feature map F of the j-th agent. j Transformed into the object feature map F of the i-th agent i Transformation characteristics M with the same coordinate system j→i It can be expressed as the following formula: M j→i =f transform (F j ,ξ’ j→i ), Where f transform This represents the action performed by the agent's transformation module; The generated feature map of the fused object is represented by the following formula: F′ i =f fusion ({M j→i } j＝1,2,…,N ), F′ i f represents the feature map of the fused object after the i-th agent incorporates features from other agents. fusion This represents the actions performed by the fusion module of the intelligent agent; and The bounding boxes of the collaborative objects are generated based on the feature maps of the fused objects, and are expressed as follows: B′ i =f decoder (F′ i ), Where B′ i Represents the bounding box of the collaborative object, f decoder This indicates the action performed by the decoding output module of the intelligent agent.

6. The hybrid collaborative 3D object detection method according to claim 5, characterized in that, Generating a fused object feature map through multi-scale fusion includes the following steps: in Represents the cooperative characteristics at the l-th spatial scale, g l Let fuse represent the residual layer with a downsampling coefficient of 2, and u represent the attention operation along the agent dimension. L represents the transpose convolution upsampling operator relative to the l-th scale, and concat represents the concatenation operator along the feature channel dimension.

7. A hybrid collaborative 3D object detection system, characterized in that, It includes multiple agents, wherein the multiple agents are configured to detect 3D objects and generate collaborative object bounding boxes, the agents including: The detection module is configured to detect 3D objects to generate agent pose, object feature map, and object bounding box; The pose correction module is configured to align multiple object bounding boxes generated by multiple agents detecting the same 3D object in order to correct the relative pose of the agents. The transformation module is configured to transform the object feature maps of multiple agents to the same coordinate system based on the corrected relative poses between the agents. The fusion module is configured to generate a fused object feature map based on the corrected relative poses between multiple agents; and The decoding output module is configured to generate a collaborative object bounding box based on the fused object feature map; The detection module also estimates the uncertainty of the bounding box, including the following steps: The boundary box B i Parameterization, expressed as follows: Where b represents the parameterized bounding box, Indicates the 3D center position of the bounding box. Indicates the length of the bounding box, Indicates the width of the bounding box, Indicates the height of the bounding box, Indicates the yaw angle of the bounding box, The variance of the center position of the bounding box, and This represents the yaw angle variance of the bounding box; Model the center positions x and y and calculate the loss L with respect to the center positions x and y. x and L y ; yaw angle Model and calculate the yaw angle. loss L θ ;as well as The summation loss function is the total loss L for training the agent to detect and overcome uncertainties. total It can be expressed as the following formula: L total =L cls +a reg L reg +a center (L x +L y )+a θ L θ , Where L cls L represents the cross-entropy loss for target classification. reg It is the loss from smoothing L1 box parametric regression, α reg α center α θ This represents the hyperparameters used for balancing.

8. A computer system, characterized in that, include: A processor, which is configured to execute machine-executable instructions; as well as A memory having machine-executable instructions stored thereon, which, when executed by a processor, perform the steps of the hybrid cooperative 3D object detection method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, performing the steps of the hybrid cooperative 3D object detection method according to any one of claims 1-6.

Images