Intelligent machine group collaboration method and system based on collaboration diagram

Abstract

The invention discloses an intelligent machine group collaboration method and system based on a collaboration diagram, and is applied to the field of intelligent machine group collaboration and vehicle-road intelligent collaboration. According to the system, a sensing processing module, a cooperative control module and a cooperative cognition module are deployed on each intelligent machine participating in cooperation, and the cooperative control module comprises a cooperative diagram generation and management module, a cooperative diagram-based cooperative control module and a cooperative diagram-based feature segmentation module. According to the method, a cooperation capability diagram is updated in real time according to sensing data, a cooperation demand diagram is generated, effective data transmission between intelligent machines is carried out through a cooperation order diagram, the intelligent machines only focus on processing and transmission of cooperation feature data, and a demand side carries out multi-source feature fusion to obtain comprehensive features for subsequent decision making. According to the method, the collaboration diagram is innovatively introduced for interaction, the inherent limitations of a single intelligent machine on perception capability and communication resources are effectively overcome through a mode of screening and transmitting high-value sparse features, and the method has advantages in a communication limited scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent machine swarm cooperation, designs vehicle-road intelligent cooperation technology, and specifically relates to a method and system for intelligent machine swarm cooperation based on a cooperation graph. Background Art

[0002] With the rapid development of artificial intelligence and Internet of Things technologies, intelligent machine swarms demonstrate powerful cooperation capabilities and adaptability in various application scenarios. An intelligent machine swarm consists of multiple intelligent machines with sensing, decision-making, and execution capabilities. These intelligent machines complete complex tasks together through efficient communication and cooperation mechanisms, improving the overall performance and reliability of the system. During the cooperation process of an intelligent machine swarm, as an effective modeling tool, a cooperation graph can clearly describe the relationships and information flows among intelligent machines, thereby optimizing the swarm cooperation strategy and enhancing the overall cooperation efficiency.

[0003] In the development process of intelligent machines, individual intelligent machines such as autonomous vehicles and drones need to possess sensing, cognitive, decision-making, and control capabilities. However, the sensing ability of an individual intelligent machine is often limited by the type of sensors, deployment locations, and environmental factors, resulting in a significant decrease in the effectiveness of sensing in complex or occluded environments. This limitation significantly increases the risks when intelligent machines execute tasks. To solve these problems, the cooperative sensing technology between intelligent machines has emerged. Through data sharing and communication between intelligent machines, cooperative sensing significantly expands the sensing range and capabilities of an individual intelligent machine. This technology allows multiple sensor platforms to work together to provide a more comprehensive view of the environmental perception, enabling intelligent machines to more accurately identify and respond to changes in the surrounding environment. For example, in an intelligent transportation system, vehicle-road cooperation is one of the important scenarios for intelligent machine swarm cooperation applications.

[0004] However, the emerging field of collaborative perception among intelligent machines still faces many challenges. First, the transmission of sensor perception information among intelligent machines depends on the support of wireless communication networks such as 5G. However, the existing 5G communication networks are difficult to meet the requirements of network communication in high-concurrency scenarios. In addition, the existing collaborative perception strategies are mainly divided into three categories according to the different transmission contents: pre-fusion (data-level fusion), middle-fusion (feature-level fusion), and post-fusion (target-level fusion). The pre-fusion strategy fuses the sensor information of intelligent machines before data input, with almost no information loss, so the detection accuracy is relatively high; the post-fusion strategy directly fuses the target perception results of multi-source sensors. Since a large amount of effective features will be lost after target detection processing, the improvement effect of collaborative perception compared to single-body perception is limited. At the same time, when a single sensor has a false detection, an effective error correction mechanism cannot be formed. The middle feature fusion is a compromise solution between the above two modes, which not only reduces the transmission bandwidth requirements but also ensures an ideal collaborative perception effect, so it has become the current mainstream information transmission strategy.

[0005] Reference Document 1: The Chinese invention patent application with the application number CN202410715790.5, titled "Vehicle-road Multi-modal Data Instance-level Feature Transmission Collaborative Perception System, Method, and Storage Medium", was published on October 1, 2024. This solution considers achieving vehicle-road collaborative perception under bandwidth-limited conditions, using a high-confidence target discrimination mechanism to screen out multi-modal instance-level transmission features, and realizing the effective extraction and dissemination of roadside features. However, this solution mainly focuses on the vehicle-road collaborative perception system. Although it can effectively reduce the computational burden on the vehicle side, it relies on instance-level feature transmission and does not fully consider the situation where the target is blocked or outside the vehicle's perception range. This leads to the fact that in actual driving, when the target is in a blocked or blind area, the single-vehicle system cannot obtain sufficient information, thus affecting the overall driving safety.

[0006] Reference Document 2: The Chinese invention patent with the application number CN202111353055.7, titled "A Multi-vehicle Collaborative Environment Perception Method Based on Semantic-level Information Fusion", was published on February 25, 2022. This solution takes into account the uncertainty of semantic information at the single-vehicle end and the influence of the environment on the perception algorithm, evaluates the confidence of the output results of the perception algorithm based on deep learning technology, and realizes the acquisition of multi-vehicle collaborative perception information based on semantic-level information fusion. This solution focuses on multi-vehicle collaborative perception based on semantic-level information fusion. Although it improves the accuracy of information acquisition, directly sending vehicle-road collaborative data to the vehicle may significantly increase the communication burden and cause processing delays. In addition, if all vehicles process the same traffic elements, it will lead to the repeated use and waste of computing resources, reducing the overall efficiency of the system.

[0007] Reference Document 3: The Chinese invention patent application with the application number CN202111013297.1 disclosed "A vehicle control system, roadside equipment, and vehicle-road collaborative system" on November 30, 2021. This solution obtains vehicle-road collaborative data transmitted by a target device in the driving environment of a target vehicle through a vehicle-road collaborative terminal. The target vehicle fuses the vehicle-road collaborative data and target perception data, and controls the driving state of the target vehicle according to the fusion result. When evaluating driving risks, this solution mainly considers the risks of the vehicle itself under the condition of driving in the same direction, and fails to comprehensively evaluate various risks in multi-vehicle and complex environments. This limitation may lead to insufficient ability of the vehicle to respond to emergencies in complex traffic environments, thereby affecting driving safety and the robustness of the system.

[0008] Reference Document 4: The Chinese invention patent application with the application number CN202310681344.2 disclosed "A multi-vehicle collaborative perception method, device, and storage medium based on lidar" on October 17, 2023. This solution shares data through operations on the original point cloud to achieve multi-vehicle collaborative perception. In this solution, the collaborative vehicle uses lidar to collect original point cloud data, performs target detection through a deep learning model (such as VoteNet), and then transmits the detection results to the vehicle itself. The collaborative vehicle performs ground segmentation, isolates obstacle point clouds, and sends key obstacle point cloud information to the vehicle itself. The vehicle itself receives the pose information and point cloud data of the collaborative vehicle, first performs rough registration processing to roughly correct the data, and then performs fine registration to optimize data alignment. Through the fused point cloud data, the vehicle itself uses a deep learning model to perform more accurate target detection and tracking, and outputs the final collaborative perception result. Although this solution expands the perception range and improves the accuracy of target detection, it relies on a cloud remote service center for data processing and collision risk assessment, and there are obvious communication delay problems. Especially when the data volume is large and the scene is complex, the demand for computing resources is extremely high, which easily leads to processing delays and affects the real-time requirements. Summary of the Invention

[0009] The following problems exist in the above technologies: Although the existing technical solutions based on intermediate feature transfer consider the feature fusion situation under different working conditions, they only perform overall compression on the transmitted feature content on the premise of an effective transmission bandwidth, lacking research on the screening and effectiveness identification of the transmitted content; since the transmitted content contains a large amount of environmental information irrelevant to the perception target, there are major defects in the density of transmitted effective information. Based on this, the present invention selects to optimize the intelligent machine group collaboration scenario on the basis of the intermediate feature sharing strategy, and proposes a method and system for intelligent machine group collaboration based on a collaboration graph. It innovatively introduces a collaboration graph for interaction, and effectively overcomes the inherent limitations of individual intelligent machines in perception ability and communication resources by screening and transmitting high-value sparse features.

[0010] An intelligent machine group collaboration system based on a collaboration graph provided by the present invention deploys a perception processing module, a collaboration control module, and a collaboration cognition module on each intelligent machine participating in the collaboration. The perception processing module obtains raw perception data from various perception devices, performs voxelization preprocessing on the point cloud data collected by the intelligent machine, and performs cleaning and time synchronization processing on the obtained perception data.

[0011] The collaboration control module includes a collaboration graph generation and management module, a collaboration control module based on the collaboration graph, and a feature segmentation module based on the collaboration graph. The collaboration graph generation and management module receives the perception data transmitted by the perception processing module in real time, and generates and updates the collaboration ability graph. The way to obtain the collaboration ability graph is as follows: for the voxelized point cloud data currently perceived by the intelligent machine, perform a three-dimensional to two-dimensional projection, calculate the two-dimensional voxel density. The greater the density, the stronger the perception ability of the intelligent machine in the corresponding area. Compare the two-dimensional voxel density with the preset density threshold A. If it is less than the density threshold A, the value of the corresponding area plane is set to 0, otherwise it is set to 1 to obtain the collaboration ability graph. The collaboration control module based on the collaboration graph calculates the collaboration demand graph according to the perception data of the current intelligent machine, and performs collaborative control between intelligent machines according to the set collaboration control mode. The collaboration order graph is obtained by taking the intersection of the collaboration demand graph of the demand side and the collaboration ability graph of the collaboration side. The way to obtain the collaboration demand graph is as follows: for the voxelized point cloud data currently perceived by the intelligent machine, perform a three-dimensional to two-dimensional projection, calculate the two-dimensional voxel density, compare the two-dimensional voxel density with the preset density threshold B. If it is less than the density threshold B, the value of the corresponding area plane is set to 1, otherwise it is set to 0 to obtain the collaboration demand graph. The collaboration control mode includes two types: one is the demand side active request mode. In this mode, the intelligent machine acting as the demand side actively broadcasts its own collaboration demand graph externally to seek collaboration with intelligent machines with corresponding perception capabilities; the other is the demand side passive acceptance mode. In this mode, the intelligent machine acting as the collaboration side actively broadcasts its own collaboration ability graph and coverage range for other intelligent machines to judge whether collaboration is needed. The feature segmentation module based on the collaboration graph realizes: on the one hand, the demand side scales the collaboration order graph according to the spatial size of the features extracted from the perception data to determine the ordered collaboration feature data, which records which dimensions of collaboration features the collaboration side needs to send; on the other hand, the collaboration side segments the features extracted at the current time step according to the received ordered collaboration feature data, and sends the segmented features to the demand side through the collaboration control module based on the collaboration graph.

[0012] The collaboration cognition module is provided with a feature extraction module and a feature fusion module. The feature extraction module extracts features from the perception data of each frame of its own, and at the same time scales and converts the collaboration features of the collaboration side into sparse features; the feature fusion module fuses the features extracted at the current time step with the collaboration features. The demand side uses the fused features for subsequent decision-making.

[0013] When the intelligent machine is set to the active request mode of the requester, the collaborative control process between the requester and the collaborator includes: within time step t, the requester calculates a collaborative requirement graph based on the perception data of the current time step and publishes the collaborative requirement graph; the collaborator calculates a collaborative capability graph based on the perception data of the current time step, calculates a collaborative graph response after receiving the collaborative requirement graph of the requester, and feeds it back to the requester; the requester calculates a collaborative order graph based on the collaborative graph response fed back by the collaborator, determines the ordered collaborative feature data and sends it to the collaborator; within time step t + 1, after receiving the ordered collaborative feature data sent by the requester, the collaborator starts to process the collaborative feature data, extracts features from the perception data of time step t + 1, performs feature segmentation according to the ordered collaborative feature data, and sends the segmented collaborative features to the requester; the requester scales the received collaborative features and fuses them with the features extracted by itself within time step t + 1; within each time step, both the requester and the collaborator repeat the collaborative control process within the above time step.

[0014] When the intelligent machine is set to the passive acceptance mode of the requester, the collaborative control process between the requester and the collaborator includes: within time step t, the collaborator calculates a collaborative capability graph based on its own current perception data and publishes it; the requester calculates a collaborative requirement graph based on its own current perception data, calculates a collaborative order graph after obtaining the collaborative capability graph of the collaborator, determines the ordered collaborative feature data and sends it to the collaborator; within time step t + 1, the collaborator extracts features based on its own current perception data, performs feature segmentation according to the ordered collaborative feature data of the requester, and sends the segmented collaborative features to the requester; the requester extracts features from its own current perception data, scales the received collaborative features, and fuses all the received collaborative features with its own features. Within each time step, both the requester and the collaborator repeat the collaborative control process within the above time step.

[0015] Corresponding to the above system, the present invention also provides a collaborative method for intelligent machine groups based on a collaborative graph, and the implementation includes the following steps:

[0016] Step 1: Deploy a perception processing module on the intelligent machine to process the data collected by the perception devices of the intelligent machine. Among them, perform voxelization preprocessing on the collected point cloud data, and perform cleaning and time synchronization processing on the obtained perception data;

[0017] Step 2: Set a collaborative control module on each intelligent machine to identify its own blind area through the collaborative control module, set the collaborative graph interaction mode, and perform feature segmentation;

[0018] The collaborative control module calculates the point density of two-dimensional voxels, compares it with a set threshold to generate a blind area mask, and the blind area mask describes the blind area of the intelligent machine in environmental perception; the collaborative control module obtains the collaborative demand map of the intelligent machine itself from the blind area mask;

[0019] The collaborative control module calculates the collaborative ability map, and the calculation method is as follows: for the voxelized point cloud data currently perceived by the intelligent machine, perform a three-dimensional to two-dimensional projection, calculate the two-dimensional voxel density, and the greater the density, the stronger the perception ability of the intelligent machine in the corresponding area. Compare the two-dimensional voxel density with a preset density threshold A. If it is less than the density threshold A, set the value of the corresponding area plane to 0, otherwise set it to 1 to obtain the collaborative ability map;

[0020] The collaborative control module takes the intersection of the collaborative demand map of the demand side and the collaborative ability map of the collaborative side to obtain the collaborative order map;

[0021] The collaborative control module sets the collaborative map interaction mode, which means realizing the collaborative control of the demand side and the collaborative side based on the collaborative demand map, collaborative ability map, and collaborative order map; there are two collaborative control modes: the demand side active request mode and the demand side passive acceptance mode;

[0022] The collaborative control module performs feature segmentation, which means that the demand side determines which dimensions of collaborative features to order from the collaborative side according to the collaborative order map; the collaborative control module performs upsampling or downsampling operations on the collaborative order map according to the spatial size of each scale feature for the multi-scale features extracted by the demand side from the perception data, so that the collaborative order map matches the size of the scale feature, and determines the ordered collaborative feature data at this scale according to the matched collaborative order map;

[0023] Multiply the matched collaborative order map element by element with the scale feature to obtain the ordered collaborative feature data at this scale;

[0024] Step 3: The intelligent machine extracts multi-scale features from the current perception data and performs feature fusion after receiving the collaborative features sent by the collaborative side as the demand side.

[0025] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0026] (1) The core innovation of the system and method of the present invention lies in introducing a collaboration graph as a medium for collaboration among intelligent machines. When an intelligent machine (the requester) discovers a blind spot in environmental perception or requires additional perceptual data support through the sensors it carries, the system encodes this demand information into a collaboration demand graph. At the same time, other intelligent machines with corresponding capabilities (the collaborators) generate a collaboration capability graph, showing the perceptual coverage range, perceptual data credibility, etc. that they can provide. By comparing the collaboration demand graph with the collaboration capability graph, the collaboration objects are intelligently matched, and the responses and feedback information during the collaboration process are recorded through collaboration responses. Finally, a collaboration order graph is generated to clarify the characteristic data required by the requester. Based on the characteristic data of the collaborator obtained from the above collaboration process, the requester extracts features from the perceptual data of the next frame and fuses multi-source features in the feature fusion module. This fusion makes full use of the complementary advantages of the features after the fusion of multiple intelligent machines, and uses the latest frame of perceptual data, which can timely reflect the current perceptual situation and avoid the delay problem that may be caused by using the previous frame of old data. The fused features synthesize the unique perceptual and processing capabilities of each intelligent machine, with higher accuracy and robustness, providing reliable, more timely and comprehensive data support for subsequent decision-making tasks.

[0027] (2) The present invention proposes an intelligent machine group collaboration system based on a collaboration graph. The design framework covers multiple key links such as perceptual processing, collaborative control, collaborative cognition, and decision-making processing, building a unified and efficient collaboration foundation for various intelligent machine groups such as intelligent connected vehicle clusters. In the collaboration system of the present invention, the collaboration graph is used as the core tool. Through the interaction of the collaboration demand graph, the collaboration capability graph, and the collaboration order graph, the perceptual requirements of the requester intelligent machine and the capability feedback of the collaborator intelligent machine are accurately described. The requester orders characteristic data from the collaborator through the collaboration order graph, and the collaborator then efficiently transmits the optimized key features back to the requester through sparse feature transmission and conducts multi-source feature fusion, realizing efficient collaborative perception and decision-making.

[0028] (3) The system and method of the present invention demonstrate significant advantages in terms of perception, collaboration, and computing resource utilization. On the one hand, the dynamic management mechanism of the collaboration graph greatly improves the flexibility of collaboration, enabling the system to quickly adapt to the changing requirements in different scenarios; on the other hand, the sparse feature transmission strategy significantly reduces the occupancy of communication bandwidth, and at the same time, by optimizing the feature transmission process, reduces data redundancy and effectively alleviates the pressure on computing resources. In addition, the clear planning of the collaboration graph in feature exchange and fusion ensures that the system and method of the present invention can operate stably and maintain efficient information flow in a high-density and multi-task collaboration environment. Especially in the typical application scenarios of intelligent connected vehicles, the system and method of the present invention improve the perception ability and safety of vehicles through the collaborative optimization of blind spot perception, providing a more solid technical support for intelligent decision-making in complex traffic environments.

[0029] (4) The system and method of the present invention design two flexible collaboration modes to adapt to the collaboration requirements in different scenarios, and solve the problem of limited perception of a single intelligent machine through the collaborative perception of the demanding intelligent machine and the collaborative intelligent machine; by designing the detailed collaboration processes of the demanding party and the collaborative party under the two collaboration modes, a systematic and standardized solution is provided for the collaborative work among intelligent machine groups; by the demanding intelligent machine generating a collaboration requirement graph in real time and interacting with the collaborative intelligent machine, the collaborative party preferentially transmits the feature information corresponding to the collaboration requirement graph to the demanding party through the network, improving the perception ability of the demanding party. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is the implementation architecture diagram of the collaborative system of intelligent machine groups based on the collaboration graph of the present invention;

[0031] Figure 2 is the schematic diagram of the collaboration process from the perspective of the demanding party when the demanding party actively requests in the present invention;

[0032] Figure 3 is the schematic diagram of the collaboration process from the perspective of the collaborative party when the demanding party actively requests in the present invention;

[0033] Figure 4 is the schematic diagram of the collaboration process from the perspective of the demanding party when the demanding party passively accepts in the present invention;

[0034] Figure 5 is the schematic diagram of the collaboration process from the perspective of the collaborative party when the demanding party passively accepts in the present invention;

[0035] Figure 6 is the implementation framework diagram of the blind spot collaborative perception method based on the collaboration graph of the present invention;

[0036] Figure 7 is the schematic diagram of the blind spot;

[0037] Figure 8 It is a diagram showing the implementation of the multi-scale feature extraction module used in the embodiments of the present invention. Detailed implementation manners

[0038] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0039] An intelligent machine refers to an entity with artificial intelligence computing capabilities that can perform perception, cognition, and decision-making tasks. Each individual intelligent machine has complete perception-cognition-decision-making capabilities. However, in complex actual application scenarios, the capabilities of a single intelligent machine are often difficult to meet the task requirements. In the collaborative perception process of a group of intelligent machines, how to efficiently utilize the perception perspectives and feature information of different intelligent machines to make up for the perception blind spots of individual intelligent machines is the key to improving the overall perception accuracy and efficiency. For example, in a vehicle-road collaborative environment with roadside devices, by transmitting features using the roadside devices and only transmitting the blind spot features of the vehicle itself, the communication pressure can be reduced.

[0040] The present invention implements a collaborative system for a group of intelligent machines based on a collaboration graph, considering collaborative intelligent machines such as fixed perception units on the roadside and demanding intelligent machines such as intelligent vehicles. Due to their characteristics and limitations of their own sensors, demanding intelligent machines need other intelligent machines to transmit data for supplementation at some blind spot positions. The present invention uses a collaboration graph to describe the collaboration between intelligent machines. The collaborative system for a group of intelligent machines based on a collaboration graph constructed by the present invention can dynamically describe and manage the relationships between various intelligent machines, and then extract, segment, and transmit feature information in a targeted manner.

[0041] As Figure 1 shown, the collaborative system for a group of intelligent machines based on a collaboration graph in the embodiments of the present invention mainly includes three core modules: a perception processing module, a collaborative control module, and a collaborative cognition module. The perception processing module is responsible for interacting with the external environment and receiving and processing the original perception data from various hardware devices. The collaborative control module realizes the collaboration between intelligent machines through the generation, management, and control of the collaboration graph. The collaborative cognition module is responsible for feature extraction and fusion to realize the comprehensive processing of multi-source data.

[0042] The perception processing module is an important bridge for the system to interact with the external environment. It receives raw data from various hardware perception devices such as cameras and lidar, including information such as images, point clouds, and radar echoes. These raw perception data, as the basic input of the system, need to undergo preliminary screening and synchronization processing. Preliminary screening is to clean up redundant and irrelevant data in the raw perception data, and synchronization processing is to synchronize the multi-source perception data collected in terms of time to ensure the timeliness and accuracy of the data. At different time steps during the system operation, the perception processing module continuously processes newly acquired raw perception data and transfers the processed data to the collaborative control module, providing the necessary data basis for subsequent collaborative graph calculation.

[0043] The collaborative control module realizes the efficient collaboration between intelligent machines through its three core sub-modules: The collaborative graph generation and management sub-module is responsible for receiving perception data and generating and updating the collaboration ability graph, providing basic collaboration information support for the entire collaboration process; The collaborative control sub-module based on the collaborative graph serves as the decision-making center, achieving effective collaborative control by calculating the collaboration demand graph and managing the interaction with the collaboration parties; The feature segmentation sub-module based on the collaborative graph focuses on the processing and transmission of collaborative feature data, ensuring that the feature data can be correctly segmented and used for subsequent feature scaling and fusion processing. These three sub-modules work closely together and cooperate to build a complete collaborative control system from collaborative graph generation, collaborative control to feature processing.

[0044] The collaborative cognition module realizes data processing and integration through the collaborative work of two sub-modules: feature extraction and feature fusion. The feature extraction sub-module is responsible for processing the perception data of each frame and extracting key features, and at the same time scaling and converting the feature data of the collaboration parties into sparse features; while the feature fusion sub-module performs multi-source fusion of the features extracted in the current time step and the collaborative features, thus realizing the comprehensive processing of multi-source data.

[0045] Figure 1 It shows an overall architecture of the collaborative system of the present invention. In the figure, "intelligent machine (demander)" represents the party that needs to improve its perception ability, which can be, for example, an intelligent vehicle, and "intelligent machine (collaborator)" represents other nodes that can provide auxiliary features, which can be, for example, roadside devices with fixed-deployment sensors, other intelligent vehicles. The collaborative strategies of different intelligent machines are controlled through the collaborative graph and can be dynamically adjusted according to the scenario, realizing an efficient closed-loop of perception, feature extraction, collaboration, and fusion.

[0046] There are two ways of collaborative control based on the collaboration graph in the system of the present invention. In different scenarios, the system can adopt appropriate collaborative strategies. One way is that the intelligent machine actively broadcasts its own sensing capabilities and coverage range outward, enabling other intelligent machines to judge whether they can make up for its sensing blind spots. This can be described as a passive acceptance strategy for the demand side. Another way is that the intelligent machine actively broadcasts its own sensing blind spot information outward, so as to seek collaborative responses from other intelligent machines with corresponding capabilities. This can be described as an active request strategy for the demand side.

[0047] As Figure 2 and Figure 3 shown, it is the implementation process of agent collaborative control under the active request of the demand side. Figure 2 and Figure 3 respectively correspond to the perspective of the demand side and the perspective of the collaborative side. The active request strategy of the demand side is that the demand side actively requests collaboration from the collaborative side at each time step, calculates the collaborative demand graph and publishes it to the collaborative side. The collaborative side generates a collaborative response according to the collaborative demand of the demand side, and sends back the collaborative feature data to the demand side for multi-source fusion, forming a closed-loop continuous collaboration process.

[0048] As Figure 2 shown, from the perspective of the demand side, the collaboration process is as follows: At time step t, the demand side first processes the original sensing data through the sensing processing module, that is Figure 2 Step 1. These processed data are sent to the collaboration graph generation and management module, that is Step 2, where the collaboration capability graph is calculated, that is Step 3. Subsequently, the collaboration capability graph is published to the collaboration control module based on the collaboration graph, that is Step 4. After receiving the collaboration capability graph, the collaboration control module calculates the collaboration demand graph, that is Step 5, and publishes the collaboration demand graph, that is Step 6. After receiving the collaboration demand graph, the collaborative side calculates its own collaboration capability graph, that is Step 7, and calculates the collaboration graph response in combination with the collaboration demand graph of the demand side, that is Step 8. The demand side calculates the collaboration order graph based on the collaboration response of the collaborative side, that is Steps 9 and 10, and sets the size of the collaboration order graph in the feature segmentation module based on the collaboration graph, that is Step 11. According to the set collaboration order graph, the ordered collaboration feature data can be obtained, that is which dimensions of feature data the demand side needs the collaborative side to send, and the ordered collaboration feature data is sent to the collaborative side, that is Step 12.

[0049] At time step t+1, after the collaborating party receives the ordered collaborative feature data, it performs feature segmentation based on the collaborative graph, i.e., step 13, and sends the collaborative feature data back to the demanding party, i.e., step 14. Importantly, at this time, the demanding party has obtained a new frame of perception data as in steps 18 and 19, and has performed feature extraction as in steps 20 and 21. When the collaborative feature data arrives, the demanding party performs feature scaling on it as in step 16 to obtain sparse features as in step 17. These sparse features are subjected to multi-source feature fusion with the features at the current time step (t+1) as in step 22. This process will continue to loop, and at each new time step, the demanding party will calculate a new collaborative ability graph as in step 24 and update it as in step 25 to ensure the continuous progress of the collaborative process.

[0050] As Figure 3 shown, from the perspective of the collaborating party, the collaborative process is as follows: At time step t, the collaborating party first processes the original perception data through the perception processing module as Figure 3 step 1, and then transmits the preprocessed data to the feature extraction module as step 2. The feature extraction model performs feature extraction and sends it to the collaborative graph generation and management module, as in steps 3 and 4. The collaborative graph generation and management module calculates the collaborative ability graph as in step 5 and publishes it to the collaborative control module based on the collaborative graph as in step 6. When the demanding party sends the collaborative requirement graph as in steps 7 to 9, the collaborative control module of the collaborating party will combine the collaborative requirement graph of the demanding party and its own collaborative ability graph, calculate the collaborative graph response as in step 10, and send it to the demanding party as in step 11. The demanding party then calculates the collaborative order graph according to its own needs and sends it to the collaborating party as in steps 12 and 13.

[0051] At time step t+1, when the collaborating party receives the ordered collaborative feature data information and the collaborative order graph sent by the demanding party, as in steps 14-15, it starts a new round of collaborative feature data processing. At this time, the perception processing module has already started processing the perception data of the next time step as in steps 16-17, and the feature extraction module is also processing new features as in steps 18-19. The feature segmentation module based on the collaborative graph will segment these new feature data as in steps 20-21, and finally send the segmented feature data to the demanding party through the collaborative control module as in step 22. The demanding party performs feature scaling and multi-source feature fusion based on the collaborative graph as in steps 23-24.

[0052] Within each time step, both the demanding party and the collaborating party repeat the above collaborative control process at the current time step t and the process of collaborative feature transmission and multi-source feature fusion according to the ordered collaborative feature data of the previous time step.

[0053] As Figure 4 and Figure 5 shown, it is the implementation process of agent collaborative control under the passive acceptance of the demanding party.Figure 4 and Figure 5 respectively show the collaboration processes from the perspectives of the requester and the collaborator in the requester passive acceptance strategy. The requester passive acceptance strategy is that the collaborator actively publishes the collaboration capability graph to the requester, and the requester passively calculates the collaboration requirements based on the collaboration capability graph and orders the feature data from the collaborator. The collaborator transmits the feature data back to the requester for sparse feature extraction and fusion, and the requester updates the collaboration capability graph at each time step, passively maintaining the continuous collaboration process.

[0054] As Figure 4 shown, from the perspective of the requester, the collaboration process is as follows: At time step t, the collaborator first calculates the collaboration capability graph through its own perception data, such as Figure 4 step 1, and publishes it to the requester as step 2, and the requester obtains the collaboration capability graph published by the collaborator as step 3. The requester calculates the collaboration capability graph based on the current self-perception data and publishes it to the collaboration control module based on the collaboration graph, such as steps 4-7. The collaboration control module of the requester based on the collaboration graph calculates the collaboration requirement graph as step 8, and calculates the collaboration order graph based on the collaboration capability graph of the collaborator and makes settings, such as steps 9-10.

[0055] At time step t+1, the requester orders the feature data from the collaborator according to the collaboration order graph, and the collaborator performs feature segmentation based on the collaboration graph and sends the collaboration feature data to the requester, such as steps 11-13. The requester performs feature extraction based on the perception data of the current time step, such as steps 17-20. The feature segmentation module based on the collaboration graph receives the collaboration feature data, such as steps 14-15, and then fuses the sparse feature obtained by scaling the collaboration feature data with the feature extracted at the current time step, such as step 21. This process will continue to loop. At each new time step, the requester will calculate a new collaboration capability graph, such as step 23, and update it, such as step 24, to ensure the continuous progress of the collaboration process.

[0056] As Figure 5 shown, from the perspective of the collaborator, the collaboration process is as follows: At time step t, the collaborator first processes the original perception data through the perception processing module, such as Figure 5In step 1, the processed data is sent to the collaborative graph generation and management module. As in step 2, the collaborative ability graph is calculated here. As in step 3, subsequently, the collaborative ability graph is published to the collaborative control module based on the collaborative graph. As in step 4, after receiving the collaborative ability graph, the collaborative control module based on the collaborative graph actively shares the collaborative graph with the requester. As in step 5, the requester calculates the collaborative ability graph and the collaborative requirement graph based on its own perception data in steps 6 - 7, then calculates the collaborative order graph based on the collaborative ability graph shared by the collaborator in step 8, and determines to send the ordered collaborative feature data to the collaborator in step 9. The collaborator initiates the collaborative feature data sending process in step 10 and saves the collaborative order graph and the ordered collaborative feature data to the feature segmentation module based on the collaborative graph in step 11.

[0057] At time step t + 1, the collaborator extracts features based on the perception data of the current time step in steps 12 - 15, segments the features according to the ordered collaborative feature data in step 16, and then sends the segmented feature data to the requester in steps 17 - 18. The requester performs feature scaling and multi-source feature fusion based on the collaborative graph in steps 19 - 20.

[0058] In each time step, both the requester and the collaborator repeat the above collaborative control process in the current time step t and the process of collaborative feature transmission and multi-source feature fusion according to the ordered collaborative feature data in the previous time step.

[0059] The system of the present invention can flexibly select collaborative strategies according to scenario requirements, so as to achieve the purpose of optimizing information transmission, enhancing perception quality, and improving decision-making accuracy under different perception conditions and task requirements. To overcome the performance limitations of the sensors that can be carried by a single intelligent machine and make up for the limited communication capabilities between intelligent machines, the system of the present invention proposes a collaborative graph as a tool for abstractly representing the collaborative relationship and data interaction between intelligent machines. The collaborative graph can flexibly express information such as requirements, capabilities, and feedback, and make the collaborative work of the intelligent machine group more flexible and efficient by dynamically updating the information of the graph.

[0060] In the system of the present invention, the collaborative graph is used as the core tool. Through the interaction of the collaborative requirement graph, the collaborative ability graph, and the collaborative order graph, it accurately describes the perception requirements of the intelligent machine of the requester and the ability feedback of the intelligent machine of the collaborator. The collaborative perception between intelligent machines essentially realizes blind area collaborative perception based on the collaborative graph, that is, the proposed method for collaborative work of intelligent machine groups based on the collaborative graph in the present invention.

[0061] The method of the present invention realizes comprehensive perception and efficient processing of blind areas through an intelligent communication and data exchange mechanism based on a collaboration graph. The present invention first uses the original point cloud data of sensors of intelligent machines such as autonomous vehicles and drones, extracts and identifies their blind areas, and abstracts them into a collaboration demand graph. At the same time, a collaboration order graph is calculated based on the collaboration ability graph of collaborative intelligent machines such as other intelligent vehicles, drones, and roadside devices and its own collaboration demand graph. The collaborative party completes feature segmentation and data exchange based on the collaboration order graph, and finally realizes feature fusion. Finally, the intelligent machine makes a series of decisions through the fused features. The collaboration ability graph is obtained by identifying and smoothing dense areas from the point cloud data, representing the effective perception area of the intelligent machine. The collaboration demand graph is the sparse part of the point cloud data, representing the perception blind area of the intelligent machine. The collaboration order graph is obtained by matching the collaboration demand graph of the demand side with the collaboration ability graph of the collaborative side. The collaboration order graph describes the data area and feature scale that the demand side requests the collaborative side to transmit.

[0062] Taking vehicle-road cooperation as an example, an implementation of the collaborative method for a group of intelligent machines based on a collaboration graph according to the present invention is as Figure 6 shown. In the scenario of vehicle-road collaborative perception, the main vehicle of autonomous driving first uses its own sensor data (such as lidar, camera, etc.) to perceive the surrounding environment, but there may be multiple perception blind areas. To solve this problem, the main vehicle identifies the collaboration ability graph based on its own perception data, and then obtains the corresponding collaboration demand graph. The collaboration demand graph represents the perception blind area or sparse area. The main vehicle sends this information, its collaboration demand graph, location and other data to the intelligent roadside unit as the collaborative party through wireless communication. The intelligent roadside unit uses the high-precision sensors (such as cameras, radars, lidars, etc.) deployed by itself to collect data on the surrounding environment to create a collaboration ability graph. The collaborative party calculates the collaboration order graph for the demand side based on its own collaboration ability graph. After the main vehicle confirms the collaboration order graph, it orders the feature data of the collaborative party. After the collaborative party performs feature segmentation, it returns sparse features. After receiving the sparse features, the main vehicle performs scaling and multi-source feature fusion based on attention. After obtaining the fused features, the main vehicle performs decision-making processing. When the roadside unit actively shares data, it not only pushes the environmental data to nearby autonomous vehicles in real time, but also optimizes the relevance and accuracy of these data through the collaboration graph, helping the vehicle to more comprehensively cover and compensate its blind areas, thereby improving the safety and efficiency of driving. On the contrary, when the vehicle shares its blind area data with the roadside unit, the roadside unit integrates and analyzes this information and feeds back more accurate blind area compensation data to the vehicle, optimizing the vehicle's perception range and decision-making ability. This two-way dynamic data exchange and collaborative working mode significantly improves the accuracy and efficiency of blind area perception, providing support for the safe and smooth operation of the intelligent transportation system.

[0063] AsFigure 6 As shown in Figure 6 , the following describes the perception processing, collaborative control, and collaborative cognition in three parts respectively.

[0064] (1) Perception processing. Perception processing refers to that an intelligent machine receives raw data from hardware perception devices such as cameras and lidar, and performs preliminary screening and synchronization processing to ensure the timeliness and accuracy of the data. In the embodiment of the present invention, the perception processing module mainly processes the point cloud data from the host vehicle and roadside devices. The specific processing process includes data reception, preprocessing, screening, and synchronization.

[0065] Data reception: The host vehicle and roadside devices respectively collect point cloud data through their respective perception devices such as lidar.

[0066] Data preprocessing: Perform voxelization processing on the received point cloud data to reduce the data volume and improve the processing efficiency. In the embodiment of the present invention, with the vehicle centroid as the origin, a three-dimensional grid coordinate system is constructed, the X-axis is set to be consistent with the vehicle driving direction, the Z-axis is perpendicular to the ground, and the Y-axis is perpendicular to the vehicle driving direction. Each axis of the grid is equally divided according to a predetermined voxel size d x , d y , d z . The specific voxelization definition is as follows:

[0067]

[0068] where p x , p y , p z are the coordinates of point p on the three axes, represents the point cloud set in space, and V(x, y, z) is the voxelized coordinate of point p.

[0069] Data screening and synchronization: Screen the voxelized point cloud data, remove noise and irrelevant points, and synchronize with the time step to ensure that the data from different perception devices are aligned within the same time frame.

[0070] For the intelligent devices in the application scenario of the method of the present invention, the acquired point cloud data can be preprocessed in the above manner. For the data acquired by other perception devices such as images, preprocessing can be performed according to pre-requirements such as size and resolution. After preprocessing the data acquired by different sensors, denoising and time synchronization are performed.

[0071] (2) Collaborative control. In the present invention, aiming at the blind areas of intelligent machines, a cooperation graph is used to realize feature cooperation and interaction between intelligent machines. Therefore, this collaborative control module includes blind area recognition, interaction of the cooperation graph, and feature segmentation in three parts.

[0072] (2.1) Blind Spot Recognition. The main objective of blind spot recognition is to identify the perception blind spots from the perception data of the host vehicle itself. For this purpose, the present invention adopts a blind spot recognition method based on point cloud data. Judging the blind spot according to the density of the point cloud is an effective method because the density of the point cloud represents the amount of information and reflects the effective information content of the area.

[0073] To evaluate the point cloud density of each area, the present invention adopts a three-dimensional to two-dimensional projection method to calculate the point density ρ(x, y) of the two-dimensional voxel as follows:

[0074]

[0075] where count(V(x, y, :)) represents the number of points contained in the voxel V(x, y, :) on the (x, y) plane. According to the preset density threshold θ, the point density ρ(x, y) is binarized to generate a blind spot mask M, that is, a collaborative demand map is obtained, where:

[0076]

[0077] where M(x, y) = 1 indicates that this area is the blind spot of the intelligent machine, and M(x, y) = 0 indicates that this area is the visible area of the intelligent machine. To eliminate noise and small area errors, morphological operations such as erosion and dilation are performed on the generated mask M to optimize the quality of the mask. Subsequently, the connected region analysis technology is applied to identify independent blind spot regions. This blind spot information will be used in the subsequent collaborative perception process. A result of blind spot recognition is shown in Figure 7 which shows the position and shape of the blind spots perceived by the host vehicle, and the areas with sparse grids represent the blind spots perceived by the vehicle.

[0078] (2.2) Collaborative Map Interaction. Collaborative map interaction is the core part of the collaborative control module, responsible for generating, sharing, and processing collaborative maps between the host vehicle and roadside devices to achieve effective data transmission and feature sharing. This process includes the generation of collaborative demand maps, the sharing of collaborative capability maps, and the calculation and transmission of collaborative order maps. The specific process and formulaic description are as follows:

[0079] Generation of Collaborative Demand Map: The host vehicle, i.e., the demanding intelligent machine, generates a collaborative demand map based on its own perception data. The main purpose is to identify its perception blind spots and requires the roadside device, i.e., the collaborative intelligent machine, to provide the perception features of the relevant areas. The generation process of the collaborative demand map is the same as the generation process of the blind spot mask, which is defined as follows:

[0080] R = M;

[0081] Among them, R represents the collaboration requirement graph, and M is the blind area mask generated by the host vehicle. The collaboration requirement graph R is used to represent the blind area regions existing in the environmental perception of the requester, and the collaborator is required to provide the perception features of these regions.

[0082] Generation and sharing of collaboration ability graph: The roadside device uses its own perception data to generate a collaboration ability graph, which represents the perception feature regions that it can provide. The generation process of the collaboration ability graph is similar to that of the collaboration requirement graph, and the specific steps are as follows:

[0083] a) Preprocessing of perception data: Based on the relative pose between the collaborator and the requester, the original point cloud data of the collaborator is transformed to the requester coordinate system through a coordinate transformation matrix to ensure the spatial alignment of the perception data of both parties. Perform a three-dimensional to two-dimensional projection on the transformed point cloud in the requester coordinate system, and calculate the two-dimensional voxel density according to the same voxel division rule as the blind area recognition:

[0084]

[0085] Among them, count(V collab (x,y,:)) represents the number of points contained in the voxel V collab (x,y,:) on the (x, y) plane. Compare the density with the threshold θ1 to generate the collaboration ability graph C as follows:

[0086]

[0087] Here, ρ collab (x,y)≥θ1 indicates that the point cloud density ρ collab (x,y) of the collaborator in this region is high enough to have a reliable perception ability. In the method of the present invention, the thresholds θ1 and θ can be set according to the actual application situation. If the thresholds are set to be equal, the collaboration ability graph and the collaboration requirement graph of the intelligent machine at the same time step are complementary relationships.

[0088] b) Sharing of collaboration ability graph: The roadside device transmits the collaboration ability graph C collab to the host vehicle through the network so that it can understand which regions' perception features the roadside device can provide.

[0089] In the active request mode of the requester, as the collaborator, the intelligent machine calculates the collaboration graph response after receiving the collaboration requirement graph sent by the requester. The intelligent machine first transforms the received collaboration requirement graph to its own coordinate system, and then obtains the intersection of it and its own collaboration ability graph. As long as the intersection is not empty, the collaboration graph response is obtained and fed back to the requester. The requester calculates the collaboration order graph after receiving the collaboration graph response. In the embodiment of the present invention, the requester will calculate the collaboration order graph according to all responses and receive the supplementary data sent by all collaborators.

[0090] In the passive acceptance mode of the demand side, after the demand side obtains the cooperation ability map of the cooperation side, it converts it to its own coordinate system, and then calculates the intersection with its own cooperation demand map to calculate the cooperation order map. The calculation process of the cooperation order map in the embodiment of the present invention in this mode is as follows: After the host vehicle receives the cooperation ability map shared by the roadside device and performs coordinate system conversion, it obtains the cooperation ability map C collab , combines it with the cooperation demand map R generated by itself, and calculates the cooperation order map S. The cooperation order map represents the effective feature area that the cooperation side can transmit to the host vehicle, that is, the intersection of the cooperation ability map C collab of the cooperation side and the cooperation demand map R of the demand side, S = C collab ∩ R.

[0091] The embodiment of the present invention calculates the cooperation order map as follows:

[0092] Among them, S(x, y) = 1 means that at the position (x, y), the roadside device can provide the sensing features required by the demand side, and S(x, y) = 0 means that this position does not meet the transmission conditions.

[0093] Cooperation order map transmission: The host vehicle sends the calculated cooperation order map S back to the roadside device, indicating which areas of feature data need to be transmitted.

[0094] (2.3) Blind area feature segmentation. The host vehicle filters out the corresponding blind area features according to the cooperation order map S, that is, orders the cooperation feature data, so as to reduce the data transmission volume and improve the communication efficiency. After the roadside device obtains the ordered cooperation feature data sent by the host vehicle, it performs feature segmentation and sends the segmented features to the host vehicle.

[0095] Since the spatial dimensions of the feature map and the cooperation order map may be different, it is necessary to perform upsampling or downsampling operations on the cooperation order map S to make its size match that of the feature map. The feature extraction module of the present invention extracts multi-scale features from the sensing data, performs upsampling or downsampling operations on the cooperation order map according to the spatial size of each scale feature, so that the cooperation order map matches the size of the scale feature, and determines the ordered cooperation feature data at this scale according to the matched cooperation order map Upsample(S).

[0096] After the roadside device obtains Upsample(S), it obtains the current multi-scale feature F through the feature extraction module road , for each scale of feature Apply Upsample(S) to the feature map. The specific operation is as follows:

[0097]

[0098] Among them, Upsample(S) adjusts S to be the same as The same spatial dimensions, where ⊙ represents element-wise multiplication. In this way, only the features at the blind spot positions are retained, and the features at the remaining positions are set to zero. The feature map after masking processing represents the segmentation features at the k-th scale.

[0099] Since only the features corresponding to the blind spot need to be transmitted, the feature map has sparsity. The roadside device will further compress and encode the segmented redundant features to reduce the amount of transmitted data. The host vehicle performs a scaling transformation to sparse features after receiving the collaborative features sent by the collaborator.

[0100] (3) Cooperative cognition. Cooperative cognition includes feature extraction and feature fusion, which in this method refers to extracting point cloud feature data and multi-source feature fusion to obtain enhanced features for subsequent decision-making.

[0101] (3.1) Feature extraction. Feature extraction is to extract multi-scale features from the perception data to provide rich environmental perception information for the host vehicle. First, the PointPillars method is used to process the collected point cloud data. The PointPillars method voxelizes the point cloud data into regular three-dimensional grids and then converts this data into a two-dimensional pseudo-image format. This conversion preserves the height information and enables the data to be applicable to a two-dimensional convolutional neural network architecture, improving the computational efficiency. Next, the feature extraction network G θ (·), which is the backbone network of PointPillars, is used to perform in-depth feature extraction on the voxelized point cloud data. This network includes a series of convolutional layers and activation functions, which can effectively extract spatial and semantic information and generate the basic feature map.

[0102] To obtain multi-scale feature representations, the present invention adopts a multi-scale feature extraction module FPN θ (·), which uses a top-down structure to gradually fuse features from high-level to low-level to achieve the integration of multi-scale information. As Figure 8As shown, the embodiment of the present invention implements a three-layer multi-scale feature extraction network. By establishing lateral connections between feature maps of different scales, high-level semantic features can be transmitted to the low level, while the high-resolution detail information of low-level features is retained. This structure can be flexibly extended to more scales to meet the requirements of different tasks. During the multi-scale feature extraction process, first, the high-level feature map is upsampled by means of transposed convolution or bilinear interpolation to match the spatial size of the low-level feature map. Subsequently, the number of channels of the low-level feature map is adjusted by using 1×1 convolution, which not only unifies the channel dimension but also helps reduce noise. Then, the upsampled high-level feature map and the adjusted low-level feature map are added element by element to achieve feature fusion. Finally, 3×3 convolution is applied to these fused feature maps for further feature refinement and noise suppression, thereby enhancing the feature expression ability and information richness. After the above processing, a multi-scale feature set F is generated ego as follows:

[0103]

[0104] wherein, represents the perception data of the host vehicle, K is the number of feature layers, represents the k-th feature layer.

[0105] (3.2) Feature fusion. Feature fusion refers to fusing the perception features of the host vehicle itself with the blind area features from roadside devices to improve the overall perception ability of the environment.

[0106] a) Attention-based feature fusion. To effectively fuse multi-scale features, a fusion method based on the attention mechanism is adopted. The features of the host vehicle and the received blind area features are linearly mapped respectively to obtain query, key, and value vectors:

[0107]

[0108] where, W Q , W K , W V are trainable weight matrices.

[0109] Calculate the attention score

[0110]

[0111] where, d k is the dimension of the feature, used for scaling to prevent the value from being too large, and the superscript T represents taking the transpose.

[0112] Weight the value vector using the attention score:

[0113]

[0114] Fuse the features of the host vehicle with the weighted roadside features, which can be done using addition or concatenation (Concat) operations:

[0115]

[0116] Or

[0117]

[0118] Finally, the fused features are obtained

[0119] b) Fusion of multi-scale features. To unify the size of the feature maps for subsequent processing, the fused multi-scale features need to be adjusted. For each fused feature map Use the decoder network Decoder for upsampling to make their spatial sizes consistent:

[0120]

[0121] The decoder can consist of a series of transposed convolutional layers or upsampling layers, which can effectively restore the spatial resolution of the feature maps. Subsequently, the feature maps of all scales are concatenated in the channel dimension to form the comprehensive feature map F ego，final As follows:

[0122]

[0123] This comprehensive feature map contains information at different scales, has stronger expressive power, and can effectively improve the performance of subsequent perception tasks. Specifically, this comprehensively fused feature map F ego，final Can be used as the input for downstream tasks. For example, it can be connected to an object detection head to achieve object detection and localization in the scene, or connected to a semantic segmentation head to perform pixel-level scene understanding. In addition, this feature map can also be used for other perception tasks, such as drivable area prediction, depth estimation, etc., and the specific application depends on the task requirements and the design of the network architecture.

[0124] To prove the effectiveness of the technical solution of the present invention in enhancing the perception effect and reducing the communication bandwidth occupancy, a large-scale collaborative perception data set is collected using the Carla simulation system, and experiments are conducted on an NVIDIA RTX 3090 GPU and a 12th generation Intel(R) Core(TM) i9-12900K CPU machine. As shown in Table 1, the multi-vehicle perception is significantly higher than the single-vehicle perception performance in terms of the AP@IoU = 0.5 metric. In multi-vehicle perception, compared with the method of basic multi-vehicle perception (sharing all perception features), the method based on the collaborative graph of the present invention reduces the average communication cost by 29.5% at the cost of a 1.8% performance drop. This demonstrates the advantage of the technical solution of the present invention in communication-constrained scenarios.

[0125] Table 1 Performance comparison between the present invention and existing methods

[0126] Method Average Traffic AP@0.5 Single-Vehicle Sensing - 62.4 Multi-Vehicle Sensing 4MB 74.3 Multi-Vehicle Sensing - Based on Collaboration Graph 2.82MB 72.9

[0127] It should be noted that the present embodiment can be implemented in hardware or a dedicated circuit, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software executed by a controller, a microprocessor, or other computing devices. When aspects of the present embodiment are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, devices, systems, technologies, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, a dedicated circuit or logic, general hardware or a controller or other computing devices, or some combination thereof.

[0128] Except for the technical features described in the specification, they are all known technologies to those skilled in the art. The present invention omits the description of well-known components and well-known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation manners described in the above embodiments do not represent all implementation manners consistent with the present application. Based on the technical solution of the present invention, various modifications or deformations that can be made by those skilled in the art without creative efforts are still within the protection scope of the present invention.

Claims

1. An intelligent machine group collaboration system based on collaboration graph, characterized in that: Deploy perception processing modules, collaborative control modules, and collaborative cognition modules on each intelligent machine participating in the collaboration; The perception processing module obtains raw perception data from various perception devices, performs voxel preprocessing on the point cloud data collected by the intelligent machine, and cleans and time-synchronizes the acquired perception data; The collaborative control module includes a collaborative graph generation and management module, a collaborative graph-based collaborative control module, and a collaborative graph-based feature segmentation module; The collaboration graph generation and management module receives the perception data transmitted by the perception processing module in real time, generates and updates the collaboration capability graph; the collaboration capability graph is obtained by performing a three-dimensional to two-dimensional projection on the voxelized point cloud data perceived by the intelligent machine at the current moment, and calculating the two-dimensional voxel density. The greater the density, the stronger the perception capability of the intelligent machine in the corresponding area. The two-dimensional voxel density is compared with the preset density threshold A. If it is less than the density threshold A, the value of the corresponding area plane is set to 0, otherwise it is set to 1 to obtain the collaboration capability graph; the collaboration control module based on the collaboration graph calculates the collaboration demand graph according to the current perception data of the intelligent machine, and performs collaborative control between the intelligent machines according to the set collaborative control mode. The collaboration ordering graph is obtained by finding the intersection of the collaboration demand graph of the demander and the collaboration capability graph of the collaborator; the collaboration demand graph is obtained by performing a three-dimensional to two-dimensional projection on the voxelized point cloud data perceived by the intelligent machine at the current moment, and calculating the two-dimensional voxel density. The two-dimensional voxel density is compared with the preset density threshold A. The density threshold B is compared. If it is less than the density threshold B, the value of the corresponding area plane is set to 1, otherwise it is set to 0, and a collaborative demand graph is obtained; the collaborative control mode includes two modes: one is the active request mode of the demand side, in which the intelligent machine as the demand side actively broadcasts its own collaborative demand graph to the outside, seeking intelligent machines with corresponding perception capabilities for collaboration; the other is the passive acceptance mode of the demand side, in which the intelligent machine as the collaborator actively broadcasts its own collaborative capability graph and coverage to the outside, so that other intelligent machines can judge whether collaboration is needed; the feature segmentation module based on the collaborative graph is implemented: on the one hand, the demand side scales the collaborative ordering graph according to the spatial size of the features extracted from the perception data, and determines the ordering collaborative feature data, which records the collaborative features of which dimensions the collaborator wants to send; on the other hand, the collaborator segments the features extracted at the current time step according to the received ordering collaborative feature data, and sends the segmented features to the demand side through the collaborative control module based on the collaborative graph; The collaborative recognition module is provided with a feature extraction module and a feature fusion module. The feature extraction module extracts features from the perception data of each frame of itself and simultaneously scales and converts the collaborative features of the collaborating party into sparse features. The feature fusion module fuses the features extracted at the current time step with the collaborative features, and the demand side uses the fused features to make decisions.

2. The system according to claim 1, characterized in that The intelligent machine is set to the demand side active request mode, and the collaborative control process between the demand side and the collaborating party includes: In time step t, the demander calculates the collaboration demand graph based on its current perception data and publishes it; the collaborator calculates the collaboration capability graph based on its current perception data, calculates the collaboration graph response after receiving the demander's collaboration demand graph and feeds it back to the demander; the demander calculates the collaboration ordering graph based on the collaboration graph response fed back by the collaborator, determines the ordering collaboration feature data and sends it to the collaborator; In time step t+1, after receiving the ordered collaborative feature data sent by the demander, the collaborative party starts collaborative feature data processing, extracts features from the perception data of time step t+1, performs feature segmentation according to the ordered collaborative feature data, and sends the segmented collaborative features to the demander; the demander scales the received collaborative features and merges them with the features extracted by itself in time step t+1; In each time step, the demander and the collaborator repeat the collaborative control process in the above time step.

3. The system according to claim 1, characterized in that The intelligent machine is set to the passive acceptance mode of the demand side. At this time, the collaborative control process between the demand side and the collaborating party includes: In time step t, the collaborator calculates the collaboration capability graph based on its current perception data and publishes it; the demander calculates the collaboration demand graph based on its current perception data, and after obtaining the capability collaboration graph of the collaborator, calculates the collaboration ordering graph, determines the ordered collaboration feature data and sends it to the collaborator; In time step t+1, the collaborative party extracts features based on its current perception data, performs feature segmentation based on the collaborative feature data ordered by the demander, and sends the segmented collaborative features to the demander; the demander extracts features from its current perception data, scales the received collaborative features, and merges all received collaborative features with its own features; In each time step, the demander and the collaborator repeat the collaborative control process in the above time step.

4. The system according to claim 1 or 2, characterized in that: The collaboration diagram response is obtained by converting the accepted collaboration demand diagram into the collaboration party's coordinate system, and then obtaining the intersection with the collaboration capability diagram of the collaboration party to obtain the collaboration diagram response. If the collaboration diagram response is not empty, the collaboration party will feedback the collaboration diagram response to the demand party, otherwise no feedback will be given.

5. A method for intelligent machine group collaboration based on collaboration graph, characterized in that: The steps include: Step 1: Deploy a perception processing module on the intelligent machine to process the data collected by the perception device of the intelligent machine, wherein the collected point cloud data is pre-processed into voxels, and the obtained perception data is cleaned and time-synchronized; Step 2: A collaborative control module is set up on each intelligent machine to identify its own blind area, set the collaborative graph interaction mode, and perform feature segmentation; The collaborative control module calculates the point density of the two-dimensional voxels and compares it with the set threshold to generate a blind area mask, which describes the blind area of ​​the intelligent machine in environmental perception. The collaborative control module obtains the collaborative demand map of the intelligent machine itself from the blind area mask. The collaborative control module calculates the collaborative capability map. The calculation method is: project the voxelized point cloud data currently perceived by the intelligent machine from three dimensions to two dimensions, calculate the two-dimensional voxel density, and the greater the density, the stronger the perception ability of the intelligent machine in the corresponding area. The two-dimensional voxel density is compared with the preset density threshold A. If it is less than the density threshold A, the value of the corresponding area plane is set to 0, otherwise it is set to 1, and the collaborative capability map is obtained; The collaborative control module obtains the intersection of the collaborative demand graph of the demander and the collaborative capability graph of the collaborator to obtain the collaborative ordering graph; The collaborative control module sets the collaborative graph interaction mode to realize the collaborative control of the demander and the collaborator based on the collaborative demand graph, collaborative capability graph and collaborative ordering graph; there are two collaborative control modes: one is the demander's active request mode, in which the intelligent machine as the demander actively broadcasts its own collaborative demand graph to the outside, seeking intelligent machines with corresponding perception capabilities to collaborate; the other is the demander's passive acceptance mode, in which the intelligent machine as the collaborator actively broadcasts its own collaborative capability graph and coverage to the outside, so that other intelligent machines can judge whether collaboration is needed; The collaborative control module performs feature segmentation, which means that the demand side determines which dimensions of collaborative features to order from the collaborative side according to the collaborative ordering graph. The collaborative control module extracts multi-scale features from the perception data of the demand side, and performs upsampling or downsampling operations on the collaborative ordering graph according to the spatial size of each scale feature, so that the collaborative ordering graph matches the size of the scale feature, and determines the ordering collaborative feature data at the scale according to the matched collaborative ordering graph. Multiply the matched collaborative ordering graph by the scale feature element by element to obtain the ordering collaborative feature data at the scale; Step 3: The intelligent machine extracts multi-scale features from the current perception data and performs feature fusion after receiving the collaborative features sent by the collaborative party as the demand party.

6. The method according to claim 5, characterized in that In the step 3, at the current time step, both the demander and the collaborator extract multi-scale features based on their own perception data; then the collaborator segments its own multi-scale features based on the ordered collaborative feature data sent by the demander at the previous time step, and transmits the segmented multi-scale features to the demander as collaborative features; the demander scales the collaborative features to convert them into sparse features, and then at each scale, calculates the weights of the sparse features based on the attention mechanism, adds or connects the weighted sparse features and the current own features to obtain fused features at each scale, and then adjusts the spatial size of each scale feature to be consistent, and then splices the feature maps of all scales in the channel dimension to form a comprehensive feature map.

Images