Multi-agent collaborative sensing method, device and system based on real representation

Abstract

The invention provides a multi-agent cooperative sensing method, device and system based on real representation. The method comprises the following steps: acquiring sensing data, target real bounding box information and corresponding BEV grid position information of a multi-agent cooperative sensing system; determining local BEV features according to sensing data of the multi-agent collaborative sensing system; generating a real BEV feature according to the target real bounding box information and the BEV grid position information; using a pre-trained modal converter to convert local BEV features from a self feature space to a shared feature space of real BEV features, and using a self-supervised learning algorithm to perform heterogeneous feature alignment so as to fuse and obtain fusion features based on real representation; and performing collaborative perception according to the fusion feature based on the real representation to obtain a collaborative perception result. According to the method, supervision is carried out through real representation, the heterogeneous features of the intelligent agent can be effectively aligned, and generalization and expandability of a collaborative sensing system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cooperative perception technology, in particular to a multi-agent cooperative perception method, device and system based on real representation. BACKGROUND

[0002] Environmental perception is the core foundation of autonomous driving, providing key information for path planning and motion control of vehicles. Among them, target detection as a key task of environmental perception has always been a research hotspot in the field of autonomous driving. However, traditional vision-based target detection methods have obvious limitations and are easily affected by light changes, rain, fog and other adverse weather conditions, resulting in low safety redundancy. In recent years, with the significant improvement of information processing capability and the gradual reduction of sensor cost, laser radar has ushered in a leap-forward development due to its excellent spatial construction capability and strong anti-weather interference characteristics. The academic and industrial circles generally believe that laser radar plays a crucial role in realizing advanced intelligent driving technology. Despite this, the current application of laser radar is mainly concentrated in the single vehicle dimension, and single-view perception has many inherent shortcomings. On the one hand, the detection range of a single laser radar vehicle-mounted sensor is relatively limited, and the effect is poor at long distances; on the other hand, during the driving process of an autonomous vehicle, a large number of visual blind areas are often encountered. These problems are often important factors that cause traffic accidents and affect traffic efficiency. Therefore, the traditional single-agent perception system is difficult to effectively solve the above problems.

[0003] To overcome the limitations of single-vehicle perception systems, multi-agent cooperative perception systems have emerged as a highly potential solution. Through vehicle-to-vehicle (V2V) and vehicle-to-everything (V2X) communication technologies, multiple agents located at different positions in the same environment can communicate and exchange information, thereby constructing a unified global perception map. By sharing perception data, the perception range can be effectively expanded, and problems such as occlusion in single-agent perception can be solved. However, existing cooperative perception technologies are mostly based on the homogeneity assumption, i.e., all agents are equipped with the same sensors and perception models. This assumption is difficult to meet in practical applications, as different agents may be equipped with different types of sensors (such as laser radar, camera, etc.) and use different perception models. This heterogeneity poses challenges to perception fusion, which may cause feature misalignment and even reduce perception performance. SUMMARY

[0004] The present application aims to solve the problems in the prior art and provides a multi-agent cooperative perception method, device and system based on real representation, which can effectively align heterogeneous features of agents through real representation supervision, improving the generalization and scalability of the cooperative perception system.

[0005] The application provides a multi-agent collaborative perception method based on real representation, comprising: acquiring perception data of a multi-agent collaborative perception system, target real bounding box information and corresponding BEV grid position information; the multi-agent collaborative perception system comprises a self-agent and at least one collaborative agent; determining local BEV features according to the perception data of the multi-agent collaborative perception system; generating real BEV features according to the target real bounding box information and the BEV grid position information; converting the local BEV features from the feature space of the self-agent to the shared feature space of the real BEV features by using a pre-trained modal converter, and performing heterogeneous feature alignment by using a self-supervised learning algorithm to obtain fused features based on real representation; and performing collaborative perception according to the fused features based on real representation to obtain a collaborative perception result.

[0006] According to the multi-agent collaborative perception method based on real representation provided by the application, the real BEV features are generated by using the target real bounding box information and the BEV grid position information, and the generation of the real BEV features comprises: encoding the target real bounding box information to obtain encoded bounding boxes; and generating the real BEV features according to the encoded bounding boxes and the BEV grid position information.

[0007] According to the multi-agent collaborative perception method based on real representation provided by the application, the generation of the real BEV features is further supervised by using a 3D detection loss.

[0008] According to the multi-agent collaborative perception method based on real representation provided by the application, the heterogeneous feature alignment is performed by using a self-supervised learning algorithm, and the heterogeneous feature alignment comprises: calculating the cosine similarity of the real BEV features and the local BEV features by using contrast learning; and aligning the heterogeneous features of positive samples according to the cosine similarity by using a cross-entropy loss function; the heterogeneous features of the positive samples are the real BEV features and the local BEV features representing the same target.

[0009] According to the multi-agent collaborative perception method based on real representation provided by the application, the loss function of the multi-agent collaborative perception comprises the cross-entropy loss function and a base detection loss function.

[0010] According to the multi-agent collaborative perception method based on real representation provided by the application, the method further comprises: in the case that a new collaborative agent is added to the multi-agent collaborative perception system, freezing the parameters of the self-perception model and the collaborative perception fusion network, and only training the modal converter.

[0011] The application further provides a multi-agent collaborative perception device based on real representation, comprising: an acquisition module, configured to acquire perception data of a multi-agent collaborative perception system, target real bounding box information, and corresponding BEV grid position information; the multi-agent collaborative perception system comprises a self-agent and at least one cooperative agent; a first feature determination module, configured to determine local BEV features according to the perception data of the multi-agent collaborative perception system; a second feature determination module, configured to generate real BEV features according to the target real bounding box information and the BEV grid position information; a fusion module, configured to convert the local BEV features from a feature space of the self-agent to a shared feature space of the real BEV features by using a pre-trained modal converter, and perform heterogeneous feature alignment by using a self-supervised learning algorithm, so as to obtain fused features based on real representation; and a collaborative perception module, configured to perform collaborative perception according to the fused features based on real representation, so as to obtain a collaborative perception result.

[0012] The application further provides a multi-agent collaborative perception system, which adopts the multi-agent collaborative perception device based on real representation.

[0013] The application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the multi-agent collaborative perception method based on real representation when executing the computer program.

[0014] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the multi-agent collaborative perception method based on real representation.

[0015] The application provides a multi-agent collaborative perception method, device and system based on real representation, the method comprising: acquiring perception data of a multi-agent collaborative perception system, target real bounding box information, and corresponding BEV grid position information; determining local BEV features according to the perception data of the multi-agent collaborative perception system; generating real BEV features according to the target real bounding box information and the BEV grid position information; converting the local BEV features from a feature space of the self-agent to a shared feature space of the real BEV features by using a pre-trained modal converter, and performing heterogeneous feature alignment by using a self-supervised learning algorithm, so as to obtain fused features based on real representation; and performing collaborative perception according to the fused features based on real representation, so as to obtain a collaborative perception result. The application can effectively align heterogeneous features of agents by supervision of real representation, and improve the generalization and scalability of the collaborative perception system. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the present application or the prior art, the drawings required to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0017] Figure 1 is a flow diagram of a multi-agent collaborative perception method based on real representation provided by the present application.

[0018] Figure 2 is one of the principle diagrams of a multi-agent collaborative perception method based on real representation provided by the present application.

[0019] Figure 3 is the second principle diagram of a multi-agent collaborative perception method based on real representation provided by the present application.

[0020] Figure 4 is a comparison diagram of visualizing BEV perception results (the left figure is a visualizing effect diagram without fusion features, and the right figure is a visualizing effect diagram based on fusion features).

[0021] Figure 5 is a structural diagram of a multi-agent collaborative perception device based on real representation provided by the present application.

[0022] Figure 6 is a structural diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present application more clear, the technical solutions in the present application will be clearly and completely described in the following with reference to the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the present application.

[0024] In the related art, multi-agent collaborative perception achieves feature enhancement and information complementation of a scene target by exchanging perception information within a certain range, and improves the accuracy and stability of scene perception. In addition, collaborative perception can also enhance the effective perception range of the agent, alleviate the dependence on long-distance and high-precision perception data, and reduce the cost of sensors. Common collaborative perception tasks include three-dimensional target detection and BEV (Bird's Eye View, bird's eye view) semantic segmentation, which have important application value in vehicle-road cooperative autonomous vehicles, vehicle-vehicle cooperative autonomous vehicles, multi-robot warehouse automation systems, unmanned aerial vehicle cluster cooperative search and rescue, underwater robot cluster cooperative target detection, unmanned aerial vehicle-quadruped robot cooperative important material transportation, and space stereoscopic monitoring.

[0025] The following will be described taking the field of autonomous driving as an example: Autonomous vehicles are widely recognized as an effective means to improve road safety. However, the inherent limitations of single-vehicle perception systems, such as susceptibility to occlusion, limited sensor coverage, and challenges in long-range perception, make them face many problems. In recent years, multi-agent collaborative perception technology has been developed to address these issues in single-vehicle perception, such as vehicle-to-vehicle (V2V) and vehicle-to-everything (V2X) collaboration. In these systems, multiple agents located at different positions in the same environment share data to construct a unified global perception map, effectively expanding the performance of single-vehicle perception. However, most existing research is usually based on the assumption of modal homogeneity, i.e., agents are equipped with homogeneous sensors and perception models. However, in multi-agent collaborative systems, heterogeneity is ubiquitous, which may introduce misaligned features into perception fusion, thereby bringing collaboration challenges.

[0026] Please refer to Figure 1 , Figure 1 A flowchart of a multi-agent collaborative perception method based on real representation provided by the present application.

[0027] Please refer to Figure 2 , Figure 2 One of the principle diagrams of a multi-agent collaborative perception method based on real representation provided by the present application.

[0028] The present application provides a multi-agent collaborative perception method based on real representation, comprising: 101: obtaining perception data of a multi-agent collaborative perception system, target real bounding box information and corresponding BEV grid position information; the multi-agent collaborative perception system comprises a self-agent and at least one collaborative agent; 102: determining a local BEV feature according to the perception data of the multi-agent collaborative perception system; 103: generating real BEV features according to the target real bounding box information and the BEV grid position information; 104: converting the local BEV features from their own feature space to the shared feature space of the real BEV features by using the pre-trained modal converter, and performing heterogeneous feature alignment by using a self-supervised learning algorithm to obtain fusion features based on real representations; 105: performing collaborative perception according to the fusion features based on real representations to obtain collaborative perception results.

[0029] To solve the technical problems existing in the prior art, the present application provides a multi-agent collaborative perception method based on real representations, which adopts a bird's eye view-based perception paradigm. First, the perception data of the ego agent and the collaborative agent are obtained, including lidar point cloud data, camera images, and corresponding pose data. In addition, the real bounding box information of the target in the perception scene and the corresponding BEV grid position information are also obtained, which can be obtained through manual labeling or high-precision maps. The ego agent and the collaborative agent use their own perception models, such as PointPillar, SECOND, or EfficientNet, to determine local BEV features from the perception data. These local features reflect the agent's preliminary understanding of the environment.

[0030] To effectively align heterogeneous features, the corresponding target representation is generated using the real target bounding box, and then it is mapped to the BEV space to generate real BEV features (real bird's eye view). To effectively align heterogeneous features, different agents are configured with specific pre-trained modal converters to convert local BEV features from their own feature space to the shared feature space of the real BEV features. Based on the self-supervised learning algorithm, the difference between the local BEV features and the real BEV features is minimized to achieve feature fusion.

[0031] Finally, collaborative perception is performed according to the fusion features based on real representations, and collaborative perception results are obtained. This result not only contains the perception information of the ego agent, but also integrates the supplementary information from the collaborative agent, thereby improving the accuracy and robustness of perception. For example, in the scenarios of vehicle-road collaborative autonomous vehicles, vehicle-vehicle collaborative autonomous vehicles, etc., this method can significantly improve the perception ability of the surrounding environment and provide more comprehensive information support for decision-making.

[0032] It needs to be explained that the multi-agent collaborative perception system includes an ego agent and at least one collaborative agent, the ego agent is, for example, an autonomous vehicle (connected car 1), and the collaborative agent is, for example, other vehicles (connected car 2) or roadside units. Each agent is equipped with its own sensors and perception models (including encoders and detection heads). Specifically, the agent iObtain raw perception data from self-sensor , and input to its encoder , obtain BEV features . The agent can choose to use self-perception or collaborative perception mode according to the demand. First, train the self-perception model to ensure that the agent can still have the ability of perception detection without using collaborative perception, and the training method and parameters depend on the self-perception model. After the agent has the ability of self-perception, train the collaborative perception model, and the parameters of the self-perception model remain unchanged in this process.

[0033] In addition, according to the dynamic changes of the current environment (such as weather conditions, light changes, traffic flow, etc.), the processing strategy and feature extraction method of the perception data are dynamically adjusted to improve the robustness of the collaborative perception method in complex environments.

[0034] Efficient communication protocols and data compression techniques are used to reduce the delay and bandwidth occupation of data transmission between agents, and to ensure the real-time and efficiency of the collaborative perception process.

[0035] The method of the application introduces real representation for supervision in the collaborative perception based on intermediate fusion, to improve the generalization of the collaborative perception system to heterogeneous agents, while ensuring its scalability.

[0036] Please refer to Figure 3 , Figure 3 Figure 2 is a schematic diagram of a multi-agent collaborative perception method based on real representation provided by the application.

[0037] As a preferred embodiment, the target real bounding box information and the BEV grid position information are used to generate real BEV features, including: encoding the target real bounding box information to obtain the encoded bounding box; according to the encoded bounding box and the BEV grid position information, generating real BEV features.

[0038] In order to effectively fuse heterogeneous features, in this embodiment, the target real bounding box information is used to generate real BEV features, thereby providing a reliable feature space for heterogeneous feature alignment. First, a real feature encoder is used to encode (label formatting) the real bounding box information (target label) of each target: , Among them, LayerNorm is a normalization layer; FC is a full connection layer; represents the target real bounding box information, including the center , the length, width and height and angle of the frame.

[0039] Then the encoded bounding box needs to be mapped to the BEV feature map. Since each grid cell of the BEV feature map represents the spatial information of a specific area in the real world. Given the grid cell size of the map s , the real-world range represented by each grid and its corresponding features are calculated. For the features of the grid , its perception range can be represented as , and , where is the lower bound of the perception range of the x axis on the feature map, is the lower bound of the y axis. Therefore, all grid cells within the coverage range of the bounding box can be regarded as a feature set. In terms of the bounding box, according to the geometric relationship, the coverage range of the bounding box is obtained . The feature set of all BEV grids corresponding to the target can be found as: , where represents the feature corresponding to the BEV grid i , and satisfy: .

[0040] Then use the encoded bounding box and the corresponding grid position information as input, generate the features in these grid cells (real bird's eye view, real BEV feature) through a multi-layer perception machine, which can be represented as: , where represents the position embedding, represents the coordinates of the grid.

[0041] In addition, when encoding the target real bounding box information, combine the spatiotemporal information of the target (such as the speed, acceleration, motion trajectory, etc. of the target) to generate real BEV features containing spatiotemporal features, so as to better capture the dynamic behavior of the target.

[0042] As a preferred embodiment, it further includes: using a 3D detection loss to supervise the generation of real BEV features.

[0043] In order to ensure the effectiveness of the real BEV features, in this embodiment, the same BEV detection loss is used for supervision: , where represents the 3D detection loss.

[0044] As a preferred embodiment, a self-supervised learning algorithm is used for heterogeneous feature alignment, including: calculating the cosine similarity between the real BEV feature and the local BEV feature using contrastive learning; aligning the heterogeneous features of the positive sample pair using the cross-entropy loss function based on the cosine similarity; the heterogeneous features of the positive sample pair are the real BEV feature and the local BEV feature representing the same target.

[0045] After obtaining the true BEV features, in this embodiment, a pre-trained modality converter is used to map the local BEV features (local bird's-eye view) into a shared feature space: , in, Represents the true characteristics of a BEV. For intelligent agents i Corresponding local BEV characteristics, It is a mode converter.

[0046] By using contrastive learning, the information contained in the BEV grids corresponding to the same target in different BEV feature maps can be aligned. For the target b Each corresponding BEV grid k Calculate the cosine similarity between the real BEV features and the agent's local BEV features: , in, Representing the mesh in the feature space of the intelligent agent itself. k The corresponding target b Features This represents the target value after average pooling in the true feature space after averaging. l Its characteristics.

[0047] Using cross-entropy loss, the contrastive loss function can be expressed as: , in, B The goal b gather, K It is the target set B The corresponding bird's-eye view grid set, k belong K .

[0048] This approach ensures that features representing the same target (positive samples versus heterogeneous features) are similar, while simultaneously increasing the differences in features representing different targets (negative samples versus heterogeneous features), thus achieving heterogeneous feature alignment. Subsequently, feature fusion is performed through a modality fusion module, and the fused features based on real representations are input into the collaborative detection head for collaborative perception, yielding the collaborative perception results.

[0049] As a preferred embodiment, the loss function of the multi-agent collaborative perception includes a cross-entropy loss function and a base detection loss function.

[0050] Considering the accuracy of target detection, in the embodiment, the base detection loss, i.e., the 3D detection loss, is introduced, so the loss function of the multi-agent collaborative perception is: wherein, is a cross-entropy loss function, is a base detection loss function.

[0051] As a preferred embodiment, it further includes: in the case that a new collaborative agent is added to the multi-agent collaborative perception system, the parameters of the self-perception model and the collaborative perception fusion network are frozen, and only the modal converter is trained.

[0052] In the embodiment, in the self-perception training stage, the perception model parameters of the agent itself remain unchanged, ensuring that the self-perception and the collaborative perception do not affect each other. In the training process of the collaborative perception system, the fusion network and the detection network parameters are fixed, and only the modal converter is updated, ensuring that the collaborative perception model has scalability and can adapt to the scene of new heterogeneous agents joining the collaboration. For the new collaborative agent, only the corresponding modal converter needs to be trained, without the need to retrain the encoder and the fusion model, so that the collaborative perception system has scalability.

[0053] The present application designs an experiment based on the OPV2V dataset, which contains four agents, as shown in Table 1, agent 1 and 2 are configured with laser radar, and the perception models are SECOND and PointPillar respectively, agent 3 and 4 are configured with RGB camera, and the perception models are EfficientNet and ResNet50 respectively.

[0054] Table 1 Heterogeneous agent configuration table

[0055] ​Table 2 gives the collaborative perception effect on the OPV2V dataset, A1, A2, A3 and A4 represent agents 1~4 respectively, wherein A1 is the Ego agent, and A2, A3 and A4 are sequentially added to the collaborative perception system. It can be seen that the perception performance based on the present Ming is better than the baseline method. Since the LiDAR agent has stronger perception ability than the Camera agent, the addition of the LiDAR agent has more obvious effect on the improvement of the perception performance. STAMP performs multiple modal conversion before fusion, and since each agent still uses its own detection network to output results, the detection network of the LiDAR agent cannot effectively utilize the shared information of the Camera agent, resulting in a decrease in the performance of collaborative perception after the addition of heterogeneous agents; HEAL adopts encoder retraining, so it still has good performance when new heterogeneous agents are added. HM-ViT adopts end-to-end training, and when the number of modalities increases, heterogeneous features are difficult to be effectively fused through a network with fixed parameters, so the performance decreases significantly. In contrast, the present application uses real features for supervision and uses contrast learning to strengthen the target-related features, so it has better perception performance.

[0056] Table 2 Collaborative perception effect comparison table

[0057] Please refer to Figure 4 , Figure 4 The comparison chart of visualizing BEV perception results (the left chart is the visualized effect chart without fusion features, and the right chart is the visualized effect chart based on fusion features).

[0058] Among them, the green box represents the real target box, the red box represents the detection result, and the colored line represents the laser point cloud. Through the visualization of the un-fused agent's own BEV representation and the fused BEV representation, it can be seen that the target area BEV is strengthened, and the features irrelevant to the target are ignored in the fusion.

[0059] On the basis of the present Ming, end-to-end driving technology can be introduced, that is, the original perception information is input, and the model directly outputs executable driving decisions, so as to integrate perception, prediction and planning into one, and improve the automatic driving ability. Through a large amount of data training, the end-to-end driving technology combined with vehicle-road cooperation can determine when to perform collaborative perception by the model, and make more reasonable driving decisions by improving the perception ability.

[0060] The multi-agent collaborative perception device based on real representation provided by the present application is described below. The multi-agent collaborative perception device based on real representation described below can be correspondingly referred to the multi-agent collaborative perception method based on real representation described above.

[0061] Please refer toFigure 5 , Figure 5 A structure diagram of a multi-agent collaborative perception device based on real representation is provided.

[0062] The application further provides a multi-agent collaborative perception device based on real representation, comprising: an acquisition module 501, configured to acquire perception data of a multi-agent collaborative perception system, target real bounding box information and corresponding BEV grid position information; the multi-agent collaborative perception system comprises a self-agent and at least one cooperative agent; a first feature determination module 502, configured to determine local BEV features according to the perception data of the multi-agent collaborative perception system; a second feature determination module 503, configured to generate real BEV features according to the target real bounding box information and the BEV grid position information; a fusion module 504, configured to convert the local BEV features from a feature space of the self-agent to a shared feature space of the real BEV features by using a pre-trained modal converter, and perform heterogeneous feature alignment by using a self-supervised learning algorithm, so as to obtain fusion features based on real representation; and a collaborative perception module 505, configured to perform collaborative perception according to the fusion features based on real representation, so as to obtain a collaborative perception result.

[0063] As a preferred embodiment, the second feature determination module 503 is specifically configured to: encode the target real bounding box information to obtain encoded bounding boxes; and generate the real BEV features according to the encoded bounding boxes and the BEV grid position information.

[0064] As a preferred embodiment, the application further comprises: supervising generation of the real BEV features by using a 3D detection loss.

[0065] As a preferred embodiment, the self-supervised learning algorithm is used to perform the heterogeneous feature alignment, which comprises: calculating cosine similarity of the real BEV features and the local BEV features by using contrast learning; and making positive samples align to the heterogeneous features by using a cross-entropy loss function according to the cosine similarity; the positive samples are the real BEV features and the local BEV features representing the same target.

[0066] As a preferred embodiment, the loss function of the multi-agent collaborative perception comprises a cross-entropy loss function and a base detection loss function.

[0067] As a preferred embodiment, the application further comprises: in the case that a new cooperative agent is added to the multi-agent collaborative perception system, freezing parameters of a self-perception model and a cooperative perception fusion network, and only training the modal converter.

[0068] The multi-agent collaborative perception system provided by the present application is described below, and the multi-agent collaborative perception system described below can be referred to each other corresponding to the multi-agent collaborative perception method based on real representation described above.

[0069] The present application also provides a multi-agent collaborative perception system, which adopts the multi-agent collaborative perception device based on real representation described above to perform multi-agent collaborative perception.

[0070] Figure 5 An example of an entity structure diagram of an electronic device is shown as Figure 5 As shown, the electronic device can include a processor 501, a communications interface 502, a memory 503, and a communications bus 504, wherein the processor 501, the communications interface 502, and the memory 503 complete mutual communication through the communications bus 504. The processor 501 can invoke the logic instructions in the memory 503 to execute the multi-agent collaborative perception method based on real representation, which includes: obtaining perception data of a multi-agent collaborative perception system, target real bounding box information, and corresponding BEV grid position information; the multi-agent collaborative perception system includes a self-agent and at least one collaborative agent; determining local BEV features according to the perception data of the multi-agent collaborative perception system; generating real BEV features according to the target real bounding box information and the BEV grid position information; converting the local BEV features from the feature space of itself to the shared feature space of the real BEV features by using a pre-trained modal converter, and performing heterogeneous feature alignment by using a self-supervised learning algorithm to obtain fusion features based on real representation; and performing collaborative perception according to the fusion features based on real representation to obtain collaborative perception results.

[0071] In addition, the logic instructions in the memory 503 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or parts of the present application that essentially contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0072] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program being stored on a non-transitory computer readable storage medium, and the computer program being executable by a processor to enable a computer to perform the real representation based multi-agent collaborative perception method provided by the above method, the method comprising: obtaining perception data of a multi-agent collaborative perception system, target real bounding box information, and corresponding BEV grid position information; the multi-agent collaborative perception system comprising a self-agent and at least one collaborative agent; determining local BEV features according to the perception data of the multi-agent collaborative perception system; generating real BEV features according to the target real bounding box information and the BEV grid position information; converting the local BEV features from the feature space of the self-agent to the shared feature space of the real BEV features by using a pre-trained modal converter, and performing heterogeneous feature alignment by using a self-supervised learning algorithm to obtain fused features based on real representation; and performing collaborative perception according to the fused features based on real representation to obtain a collaborative perception result.

[0073] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the real representation based multi-agent collaborative perception method provided by the above method, the method comprising: obtaining perception data of a multi-agent collaborative perception system, target real bounding box information, and corresponding BEV grid position information; the multi-agent collaborative perception system comprising a self-agent and at least one collaborative agent; determining local BEV features according to the perception data of the multi-agent collaborative perception system; generating real BEV features according to the target real bounding box information and the BEV grid position information; converting the local BEV features from the feature space of the self-agent to the shared feature space of the real BEV features by using a pre-trained modal converter, and performing heterogeneous feature alignment by using a self-supervised learning algorithm to obtain fused features based on real representation; and performing collaborative perception according to the fused features based on real representation to obtain a collaborative perception result.

[0074] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0075] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0076] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-agent collaborative perception method based on real representation, characterized in that, The method comprises the following steps: obtaining perception data, target real bounding box information and corresponding BEV grid position information of a multi-agent collaborative perception system; the multi-agent collaborative perception system comprises a self-agent and at least one collaborative agent; determining local BEV features according to the perception data of the multi-agent collaborative perception system; generating real BEV features according to the target real bounding box information and the BEV grid position information; using a pre-trained modal converter to convert the local BEV features from the feature space of the self-agent to the shared feature space of the real BEV features, and using a self-supervised learning algorithm to perform heterogeneous feature alignment to obtain fusion features based on real representations; performing collaborative perception according to the fusion features based on real representations to obtain collaborative perception results.

2. The real-characteristics-based multi-agent collaborative perception method according to claim 1, characterized in that, The method further comprises the following steps: encoding the target real bounding box information to obtain encoded bounding boxes; generating the real BEV features according to the encoded bounding boxes and the BEV grid position information.

3. The real-characteristics-based multi-agent collaborative perception method according to claim 1, characterized in that, The method further comprises the following steps: using a 3D detection loss to supervise the generation of the real BEV features.

4. The real-characteristics-based multi-agent collaborative perception method according to claim 1, characterized in that, The method further comprises the following steps: calculating the cosine similarity of the real BEV features and the local BEV features using contrastive learning; using a cross-entropy loss function to align the positive sample pair of heterogeneous features according to the cosine similarity; the positive sample pair of heterogeneous features are the real BEV features and the local BEV features representing the same target.

5. The real-characteristics-based multi-agent cooperative perception method according to claim 4, characterized in that, The loss function of the multi-agent collaborative perception comprises the cross-entropy loss function and a base detection loss function.

6. The real-characteristics-based multi-agent collaborative perception method according to any one of claims 1 to 5, characterized in that, The method further comprises the following steps: in the case that a new collaborative agent is added to the multi-agent collaborative perception system, freezing the parameters of the self-perception model and the collaborative perception fusion network, and only training the modal converter.

7. A real representation based multi-agent collaborative perception apparatus, characterized in that, The method comprises the following steps: a obtaining module is configured to obtain perception data, target real bounding box information and corresponding BEV grid position information of a multi-agent collaborative perception system; the multi-agent collaborative perception system comprises a self-agent and at least one collaborative agent; a first feature determining module is configured to determine local BEV features according to the perception data of the multi-agent collaborative perception system; a second feature determining module is configured to generate real BEV features according to the target real bounding box information and the BEV grid position information; a fusion module is configured to use a pre-trained modal converter to convert the local BEV features from the feature space of the self-agent to the shared feature space of the real BEV features, and use a self-supervised learning algorithm to perform heterogeneous feature alignment to obtain fusion features based on real representations; a collaborative perception module is configured to perform collaborative perception according to the fusion features based on real representations to obtain collaborative perception results. 8.A multi-agent collaborative perception system, characterized in that, The method for multi-agent collaborative perception based on real representations is performed by using the multi-agent collaborative perception device based on real representations of claim 7.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to realize the method for multi-agent collaborative perception based on real representations of any one of claims 1 to 6. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the multi-agent collaborative perception method based on real representation according to any one of claims 1 to 6.

Images