A collaborative method and system for intelligent machine swarm collaboration based on collaboration graphs
By introducing a collaborative graph mechanism among intelligent machine groups to filter and transmit high-value sparse features, the problem of insufficient effective information density in collaborative perception among intelligent machine groups is solved, and efficient and flexible collaborative perception and decision support are achieved.
Patent Information
- Application Number
- CN202510221597.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-27
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2045-02-27
AI Technical Summary
Existing intelligent machine swarm collaborative perception technologies struggle to meet network communication demands in high-concurrency scenarios, and the transmitted content contains a large amount of irrelevant environmental information, resulting in insufficient effective information density and affecting perception accuracy and system robustness.
By introducing a collaborative graph mechanism, high-value sparse features are filtered and transmitted through the generation and management of collaborative graphs among intelligent machines, and the transmitted content is optimized to achieve efficient collaborative perception among intelligent machine groups.
It improves the perception accuracy and robustness of intelligent machine swarms, reduces communication bandwidth consumption and computing resource pressure, and enhances the system's flexibility and real-time performance, especially improving vehicle perception capabilities and safety in complex traffic environments.
Smart Images

Figure CN120164083B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent machine swarm collaboration technology, and designs vehicle-road intelligent collaboration technology, specifically involving an intelligent machine swarm collaboration method and system based on a collaboration graph. Background Technology
[0002] With the rapid development of artificial intelligence and the Internet of Things (IoT) technologies, intelligent machine swarms are demonstrating strong collaborative capabilities and adaptability in various application scenarios. An intelligent machine swarm consists of multiple intelligent machines possessing perception, decision-making, and execution capabilities. These machines collaborate through efficient communication and cooperation mechanisms to complete complex tasks, improving the overall performance and reliability of the system. In the collaborative process of intelligent machine swarms, collaboration graphs, as an effective modeling tool, can clearly describe the relationships and information flow between intelligent machines, thereby optimizing swarm collaboration strategies and improving overall collaborative efficiency.
[0003] In the development of intelligent machines, individual intelligent machines, such as autonomous vehicles and drones, need to possess the capabilities of perception, cognition, decision-making, and control. However, the perception capabilities of a single intelligent machine are often limited by the type of sensor, its deployment location, and environmental factors, leading to a significant decrease in the effectiveness of perception in complex or occluded environments. This limitation significantly increases the risks when intelligent machines perform tasks. To address these issues, collaborative perception technology among intelligent machines has emerged. Through data sharing and communication between intelligent machines, collaborative perception significantly expands the perception range and capabilities of a single intelligent machine. This technology allows multiple sensor platforms to work collaboratively, providing a more comprehensive view of the environment, enabling intelligent machines to more accurately identify and respond to changes in their surroundings. For example, in intelligent transportation systems, vehicle-to-infrastructure (V2I) communication is one of the important scenarios for the collaborative application of intelligent machine swarms.
[0004] However, the emerging field of collaborative sensing among intelligent machine swarms still faces many challenges. First, the transmission of sensor information between intelligent machines relies on wireless communication networks such as 5G, but existing 5G networks struggle to meet network communication demands in high-concurrency scenarios. Furthermore, existing collaborative sensing strategies are mainly divided into three categories based on the content being transmitted: pre-fusion (data-level fusion), intermediate fusion (feature-level fusion), and post-fusion (target-level fusion). Pre-fusion strategies fuse sensor information from intelligent machines before data input, resulting in almost no information loss and thus high detection accuracy. Post-fusion strategies directly fuse target perception results from multiple sensors; however, because a large number of effective features are lost after target detection processing, the improvement of collaborative sensing compared to single-machine sensing is limited, and an effective error correction mechanism cannot be formed when a single sensor makes a false detection. Intermediate feature fusion is a compromise between the two models, reducing transmission bandwidth requirements while ensuring a relatively ideal collaborative sensing effect, thus becoming the mainstream information transmission strategy.
[0005] Reference document 1: Chinese invention patent application CN202410715790.5, published on October 1, 2024, entitled "Vehicle-Road Multimodal Data Instance-Level Feature Transmission Cooperative Perception System, Method, and Storage Medium." This scheme considers achieving vehicle-road cooperative perception under bandwidth-constrained conditions, utilizing a high-confidence target discrimination mechanism to filter multimodal instance-level transmission features, thereby achieving effective extraction and propagation of roadside features. However, this scheme mainly targets vehicle-road cooperative perception systems. Although it can effectively reduce the computational burden on the vehicle side, it relies on instance-level feature transmission and fails to fully consider situations where the target is occluded or beyond the vehicle's perception range. This results in the single-vehicle system being unable to obtain sufficient information when the target is occluded or in a blind spot during actual driving, thus affecting overall driving safety.
[0006] Reference document 2: Chinese invention patent application number CN202111353055.7, published on February 25, 2022, entitled "A Multi-Vehicle Cooperative Environmental Perception Method Based on Semantic-Level Information Fusion." This solution considers the uncertainty of semantic information at the single-vehicle end and the influence of the environment on the perception algorithm, evaluates the confidence of the output results of the perception algorithm based on deep learning technology, and realizes the acquisition of multi-vehicle cooperative perception information based on semantic-level information fusion. This solution focuses on multi-vehicle cooperative perception based on semantic-level information fusion. Although it improves the accuracy of information acquisition, it directly sends vehicle-road cooperative data to vehicles, which may significantly increase the communication burden and cause processing delays. In addition, if all vehicles process the same traffic elements, it will lead to the duplication and waste of computing resources, reducing the overall efficiency of the system.
[0007] Reference document 3: Chinese invention patent application CN202111013297.1, published on November 30, 2021, discloses "A vehicle control system, roadside equipment, and vehicle-road cooperative system." This solution acquires vehicle-road cooperative data transmitted from the target equipment in the target vehicle's driving environment through a vehicle-road cooperative terminal. The target vehicle then fuses the vehicle-road cooperative data and target perception data, controlling its driving state based on the fusion result. However, this solution primarily considers the risk of the vehicle traveling in the same direction during driving risk assessment, failing to comprehensively assess various risks in multi-vehicle and complex environments. This limitation may lead to insufficient ability of the vehicle to cope with emergencies in complex traffic environments, thus affecting driving safety and system robustness.
[0008] Reference document 4: Chinese invention patent application CN202310681344.2, published on October 17, 2023, discloses "A Multi-Vehicle Cooperative Perception Method, Device, and Storage Medium Based on LiDAR". This scheme achieves multi-vehicle cooperative perception by sharing data through operations on raw point clouds. In this scheme, the cooperative vehicle uses LiDAR to collect raw point cloud data and performs target detection using a deep learning model (such as VoteNet). The detection results are then transmitted to the driver vehicle. The cooperative vehicle performs ground segmentation, isolates obstacle point clouds, and sends key obstacle point cloud information to the driver vehicle. The driver vehicle receives the pose information and point cloud data from the cooperative vehicle, performs coarse registration to roughly correct the data, and then performs fine registration to optimize data alignment. Through the fused point cloud data, the driver vehicle uses a deep learning model for more accurate target detection and tracking, outputting the final cooperative perception result. Although this scheme expands the perception range and improves the accuracy of target detection, it relies on a cloud-based remote service center for data processing and collision risk assessment, resulting in significant communication latency issues. Especially when the data volume is large and the scenario is complex, the demand for computing resources is extremely high, which can easily lead to processing delays and affect the requirements for real-time performance. Summary of the Invention
[0009] The aforementioned technologies suffer from the following problems: While existing intermediate feature-based solutions consider feature fusion under different operating conditions, they only compress the transmitted feature content as a whole within the constraints of effective transmission bandwidth, lacking research on content filtering and effectiveness identification. Furthermore, the transmission content contains a large amount of environmental information unrelated to the sensing target, resulting in significant deficiencies in the density of effective transmitted information. Therefore, this invention optimizes intelligent machine swarm collaboration scenarios based on an intermediate feature sharing strategy, proposing a collaborative graph-based intelligent machine swarm collaboration method and system. This innovatively introduces a collaborative graph for interaction, effectively overcoming the inherent limitations of individual intelligent machines in sensing capabilities and communication resources by filtering and transmitting high-value sparse features.
[0010] This invention provides a collaborative system for intelligent machine groups based on a collaboration graph. Each participating intelligent machine deploys a perception processing module, a collaborative control module, and a collaborative cognition module. The perception processing module acquires raw perception data from various sensing devices, performs voxelization preprocessing on the point cloud data collected by the intelligent machines, and cleans and synchronizes the acquired perception data over time.
[0011] The collaborative control module includes a collaborative graph generation and management module, a collaborative graph-based collaborative control module, and a collaborative graph-based feature segmentation module. The collaborative graph generation and management module receives perception data transmitted from the perception processing module in real time and generates and updates the collaborative capability graph. The collaborative capability graph is obtained by projecting the voxelized point cloud data currently perceived by the intelligent machine from 3D to 2D, calculating the 2D voxel density. A higher density indicates a stronger perception capability of the intelligent machine in the corresponding area. The 2D voxel density is compared with a preset density threshold A. If it is less than the threshold A, the value of the corresponding area plane is set to 0; otherwise, it is set to 1, thus obtaining the collaborative capability graph. The collaborative graph-based collaborative control module calculates the collaborative demand graph based on the current perception data of the intelligent machine, performs collaborative control between intelligent machines according to the set collaborative control mode, and obtains the collaborative ordering graph by finding the intersection of the demand side's collaborative demand graph and the collaborating side's collaborative capability graph. The collaborative requirement map is obtained by projecting the voxelized point cloud data currently perceived by the intelligent machine from 3D to 2D, calculating the 2D voxel density, and comparing the 2D voxel density with a preset density threshold B. If the density is less than the threshold B, the value of the corresponding region plane is set to 1; otherwise, it is set to 0, thus obtaining the collaborative requirement map. There are two collaborative control modes: one is a proactive request mode, in which the intelligent machine acting as the requester actively broadcasts its own collaborative requirement map to seek collaboration with other intelligent machines possessing corresponding perception capabilities; the other is a passive acceptance mode, in which the intelligent machine acting as the collaborator actively broadcasts its own collaborative capability map and coverage area for other intelligent machines to determine whether collaboration is needed. The feature segmentation module based on the collaborative map is implemented as follows: on one hand, the requester scales the collaborative ordering map according to the spatial dimensions of the features extracted from the perception data to determine the ordered collaborative feature data. This data records which dimensions of collaborative features the collaborator needs to send. On the other hand, the collaborator segments the features extracted at the current time step based on the received ordered collaborative feature data, and sends the segmented features to the requester through the collaborative control module based on the collaborative map.
[0012] The collaborative cognition module consists of two modules: feature extraction and feature fusion. The feature extraction module extracts features from its own perceptual data for each frame, while simultaneously scaling and converting the collaborative features from collaborating parties into sparse features. The feature fusion module merges the features extracted at the current time step with the collaborative features. The requesting party uses the fused features for subsequent decision-making.
[0013] When the intelligent machine is set to a demand-side proactive request mode, the collaborative control process between the demander and the collaborator includes: within time step t, the demander calculates a collaborative demand graph based on the perception data of the current time step and publishes the collaborative demand graph; the collaborator calculates a collaborative capability graph based on the perception data of the current time step, calculates a collaborative graph response after receiving the collaborative demand graph from the demander, and sends it back to the demander; the demander calculates a collaborative ordering graph based on the collaborative graph response from the collaborator, determines the ordering collaborative feature data, and sends it to the collaborator; within time step t+1, after receiving the ordering collaborative feature data sent by the demander, the collaborator initiates collaborative feature data processing, extracts features from the perception data of time step t+1, segments the features according to the ordering collaborative feature data, and sends the segmented collaborative features to the demander; the demander scales the received collaborative features and merges them with the features extracted by itself within time step t+1; within each time step, the demander and the collaborator repeat the collaborative control process within the above time steps.
[0014] When the intelligent machine is set to a passive receiving mode for the demand side, the collaborative control process between the demand side and the collaborator includes: within time step t, the collaborator calculates a collaborative capability map based on its current self-perceived data and publishes it; the demand side calculates a collaborative demand map based on its current self-perceived data, and after obtaining the collaborator's capability collaboration map, calculates a collaborative ordering map, determines the ordering collaboration feature data, and sends it to the collaborator; within time step t+1, the collaborator extracts features based on its current self-perceived data, segments features based on the demand side's ordering collaboration feature data, and sends the segmented collaboration features to the demand side; the demand side extracts features from its current self-perceived data, scales the received collaboration features, and merges all received collaboration features with its own features. Within each time step, the demand side and the collaborator repeat the collaborative control process described above.
[0015] Corresponding to the above system, the present invention also provides a method for intelligent machine group collaboration based on a collaboration graph, which includes the following steps:
[0016] Step 1: Deploy a perception processing module on the intelligent machine to process the data collected by the machine's perception devices. This includes voxelization preprocessing of the collected point cloud data and cleaning and time synchronization processing of the acquired perception data.
[0017] Step 2: Set up a collaborative control module on each intelligent machine. The collaborative control module identifies its own blind spots, sets up the collaborative graph interaction mode, and performs feature segmentation.
[0018] The collaborative control module calculates the point density of two-dimensional voxels and compares it with a set threshold to generate a blind spot mask. The blind spot mask describes the blind spot area in the intelligent machine's environmental perception. The collaborative control module obtains the collaborative requirement map of the intelligent machine itself from the blind spot mask.
[0019] The collaborative control module calculates the collaborative capability map by projecting the voxelized point cloud data currently perceived by the intelligent machine from three dimensions to two dimensions, calculating the two-dimensional voxel density. The higher the density, the stronger the perception capability of the intelligent machine in the corresponding area. The two-dimensional voxel density is compared with a preset density threshold A. If it is less than the density threshold A, the value of the corresponding area plane is set to 0; otherwise, it is set to 1, thus obtaining the collaborative capability map.
[0020] The collaborative control module obtains the collaborative ordering diagram by finding the intersection of the collaborative requirements diagram of the demand side and the collaborative capability diagram of the collaborators.
[0021] The collaborative control module sets the collaborative graph interaction mode, which refers to realizing the collaborative control between the demander and the collaborator based on the collaborative demand graph, collaborative capability graph, and collaborative ordering graph; there are two collaborative control modes: demander active request mode and demander passive acceptance mode.
[0022] Feature segmentation in the collaborative control module refers to the demander determining which dimensions of collaborative features to order from the collaborator based on the collaborative ordering diagram. The collaborative control module extracts multi-scale features from the perceptual data by the demander and performs upsampling or downsampling operations on the collaborative ordering diagram according to the spatial size of each scale feature, so that the collaborative ordering diagram matches the size of the scale feature. Based on the matched collaborative ordering diagram, the ordering collaborative feature data at that scale is determined.
[0023] The matched collaborative ordering graph is multiplied element by element with the feature at that scale to obtain the collaborative ordering feature data at that scale.
[0024] Step 3: The intelligent machine extracts multi-scale features from the current perceived data and performs feature fusion after receiving collaborative features sent by the collaborator as the demand side.
[0025] Compared with the prior art, the advantages and positive effects of the present invention are as follows:
[0026] (1) The core innovation of the system and method of this invention lies in introducing a collaboration graph as a medium for collaboration between intelligent machines. When an intelligent machine (the demander) discovers a blind spot in environmental perception or needs additional perception data support through its onboard sensors, the system encodes this demand information into a collaboration demand graph. At the same time, other intelligent machines (collaborators) with corresponding capabilities generate collaboration capability graphs, showing the perception coverage range they can provide, the reliability of perception data, and other information. By comparing the collaboration demand graph and the collaboration capability graph, collaboration objects are intelligently matched, and response and feedback information during the collaboration process is recorded through collaboration responses. Finally, a collaboration order graph is generated, which clarifies the feature data required by the demander. The collaborator returns the feature data based on the collaboration order graph. After obtaining the feature data of the collaborator based on the above collaboration process, the demander extracts features based on the perception data of the next frame and fuses the multi-source features in the feature fusion module. This fusion fully utilizes the complementary advantages of the features after fusion of multiple intelligent machines, and uses the perception data of the latest frame, which can reflect the current perception situation in a timely manner and avoid the delay problem that may be caused by using old data from the previous frame. The fused features combine the unique perception and processing capabilities of each intelligent machine, resulting in higher accuracy and robustness, and providing reliable, more timely, and comprehensive data support for subsequent decision-making tasks.
[0027] (2) This invention proposes a collaborative system for intelligent machine swarms based on a collaboration graph. The design framework covers multiple key aspects such as perception processing, collaborative control, collaborative cognition, and decision processing, providing a unified and efficient collaborative foundation for various intelligent machine swarms, such as intelligent connected vehicle clusters. In this collaborative system, the collaboration graph is used as the core tool. Through the interaction of the collaboration demand graph, collaboration capability graph, and collaboration ordering graph, the system accurately describes the perception needs of the demanding intelligent machine and the capability feedback of the collaborating intelligent machine. The demanding party orders feature data from the collaborating party through the collaboration ordering graph. The collaborating party then efficiently transmits the optimized key features back to the demanding party through sparse feature transmission and performs multi-source feature fusion, achieving efficient collaborative perception and decision-making.
[0028] (3) The system and method of this invention exhibit significant advantages in perception, collaboration, and computational resource utilization. On the one hand, the dynamic management mechanism of the collaboration graph greatly improves the flexibility of collaboration, enabling the system to quickly adapt to changes in requirements under different scenarios. On the other hand, the sparse feature transmission strategy significantly reduces the occupation of communication bandwidth, and at the same time, by optimizing the feature transmission process, it reduces data redundancy and effectively alleviates the pressure on computational resources. In addition, the explicit planning of the collaboration graph in feature exchange and fusion ensures that the system and method of this invention can operate stably and maintain efficient information flow in a high-density, multi-task collaborative environment. Especially in the typical application scenarios of intelligent connected vehicles, the system and method of this invention improve the perception capability and safety of vehicles through the collaborative optimization of blind spot perception, providing more solid technical support for intelligent decision-making in complex traffic environments.
[0029] (4) The system and method of this invention are designed with two flexible collaboration modes to adapt to the collaboration needs in different scenarios. The collaboration between the intelligent machine of the demand side and the intelligent machine of the collaborator solves the problem of limited perception of a single intelligent machine. By designing the detailed collaboration process between the demand side and the collaborator under the two collaboration modes, a systematic and standardized solution is provided for the collaborative work between intelligent machine groups. The intelligent machine of the demand side generates a collaboration requirement map in real time and interacts with the intelligent machine of the collaborator, so that the collaborator can transmit the feature information corresponding to the collaboration requirement map to the demand side through the network first, thereby improving the perception ability of the demand side. Attached Figure Description
[0030] Figure 1 This is an implementation architecture diagram of the intelligent machine swarm collaboration system based on a collaboration graph according to the present invention;
[0031] Figure 2 This is a schematic diagram of the collaboration process from the perspective of the demander when the demander actively requests it, according to the present invention.
[0032] Figure 3 This is a schematic diagram of the collaboration process from the perspective of the collaborating party when the demander actively requests it according to the present invention.
[0033] Figure 4 This is a schematic diagram of the collaborative process from the perspective of the demander when the demander passively accepts the terms of the present invention.
[0034] Figure 5 This is a schematic diagram of the collaboration process from the perspective of the collaborating party when the demander passively accepts the request in this invention.
[0035] Figure 6 This is a framework diagram of the blind zone collaborative perception method based on the collaboration graph of the present invention;
[0036] Figure 7 This is a diagram of the blind spot;
[0037] Figure 8 This is a diagram illustrating the implementation of the multi-scale feature extraction module used in an embodiment of the present invention. Detailed Implementation
[0038] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.
[0039] Intelligent machines are entities with artificial intelligence computing capabilities, capable of performing perception, cognition, and decision-making tasks. Each individual intelligent machine possesses complete perception-cognition-decision capabilities. However, in complex real-world application scenarios, the capabilities of a single intelligent machine are often insufficient to meet task requirements. In the collaborative perception process of a group of intelligent machines, efficiently utilizing the perception perspectives and feature information of different intelligent machines to compensate for the perception blind spots of individual intelligent machines is key to improving overall perception accuracy and efficiency. For example, in a vehicle-to-infrastructure (V2I) environment with roadside equipment, utilizing the roadside equipment to transmit features, while only transmitting the blind spot features of one's own vehicle, can reduce communication pressure.
[0040] This invention implements a collaborative graph-based intelligent machine swarm system, considering collaborating intelligent machines such as fixed sensing units on the roadside and demanding intelligent machines such as intelligent vehicles. Due to their characteristics and sensor limitations, demanding intelligent machines require data transmission from other intelligent machines in blind spots. This invention uses a collaborative graph to describe the collaboration between intelligent machines. The collaborative graph-based intelligent machine swarm system constructed by this invention can dynamically describe and manage the relationships between intelligent machines, thereby enabling targeted extraction, segmentation, and transmission of feature information.
[0041] like Figure 1 As shown, the intelligent machine swarm collaboration system based on collaboration graphs in this embodiment of the invention mainly includes three core modules: a perception processing module, a collaboration control module, and a collaboration cognition module. The perception processing module is responsible for interacting with the external environment, receiving and processing raw perception data from various hardware devices. The collaboration control module realizes collaboration between intelligent machines through the generation, management, and control of collaboration graphs. The collaboration cognition module is responsible for feature extraction and fusion, achieving comprehensive processing of multi-source data.
[0042] The perception processing module serves as a crucial bridge for the system's interaction with the external environment. It receives raw data from various hardware sensing devices, such as cameras and LiDAR, including images, point clouds, and radar echoes. This raw perception data, as the system's fundamental input, undergoes initial filtering and synchronization processing. Initial filtering removes redundant and irrelevant data, while synchronization processing synchronizes the collected multi-source perception data to ensure timeliness and accuracy. Throughout different time steps of system operation, the perception processing module continuously processes newly acquired raw perception data and transmits the processed data to the collaborative control module, providing the necessary data foundation for subsequent collaborative graph calculations.
[0043] The collaborative control module achieves efficient collaboration between intelligent machines through its three core sub-modules: the collaboration graph generation and management sub-module receives sensing data and generates and updates the collaboration capability graph, providing basic collaboration information support for the entire collaboration process; the collaboration graph-based collaboration control sub-module acts as the decision-making center, calculating the collaboration demand graph and managing interactions with collaborating parties to achieve effective collaboration control; and the collaboration graph-based feature segmentation sub-module focuses on the processing and transmission of collaboration feature data, ensuring that feature data can be correctly segmented and used for subsequent feature scaling and fusion processing. These three sub-modules work closely together to construct a complete collaborative control system from collaboration graph generation and collaboration control to feature processing.
[0044] The collaborative cognition module achieves data processing and integration through two sub-modules: feature extraction and feature fusion. The feature extraction sub-module is responsible for processing the perceptual data of each frame and extracting key features, while scaling and converting the feature data of collaborating parties into sparse features. The feature fusion sub-module, on the other hand, performs multi-source fusion of the features extracted at the current time step with the collaborative features, thereby achieving comprehensive processing of multi-source data.
[0045] Figure 1 This diagram illustrates the overall architecture of the collaborative system of this invention. In the diagram, "intelligent machine (demand side)" represents the party needing enhanced perception capabilities, such as an intelligent vehicle, while "intelligent machine (collaborator)" represents other nodes that can provide auxiliary features, such as roadside equipment with fixed-deployment sensors or other intelligent vehicles. The collaborative strategy of different intelligent machines is controlled through a collaborative graph, and can be dynamically adjusted according to the scenario to achieve an efficient closed loop of perception, feature extraction, collaboration, and fusion.
[0046] The system of this invention employs two cooperative control methods based on a cooperation graph, allowing the system to adopt appropriate cooperative strategies in different scenarios. One method involves the intelligent machine actively broadcasting its sensing capabilities and coverage area, enabling other intelligent machines to determine whether they can compensate for its sensing blind spots. This can be described as a passive acceptance strategy by the demand side. The other method involves the intelligent machine actively broadcasting its sensing blind spot information, thereby seeking cooperative responses from other intelligent machines with corresponding capabilities. This can be described as a proactive request strategy by the demand side.
[0047] like Figure 2 and Figure 3 The diagram illustrates the process of intelligent agent collaborative control implemented under the active request of the demand side. Figure 2 and Figure 3 These correspond to the perspectives of the demander and the collaborator, respectively. The demander's proactive request strategy involves the demander actively requesting collaboration from the collaborator at each time step, calculating the collaboration requirement graph, and publishing it to the collaborator. The collaborator generates a collaboration response based on the demander's collaboration requirements and sends the collaboration feature data back to the demander for multi-source fusion, forming a closed-loop continuous collaboration process.
[0048] like Figure 2 As shown, from the perspective of the demand side, the collaboration process is as follows: At time step t, the demand side first processes the raw sensing data through the sensing processing module, that is... Figure 2 Step 1: The processed data is sent to the collaboration graph generation and management module (Step 2), where the collaboration capability graph is calculated (Step 3). Subsequently, the collaboration capability graph is published to the collaboration control module based on the collaboration graph (Step 4). Upon receiving the collaboration capability graph, the collaboration control module calculates the collaboration requirement graph (Step 5) and publishes it (Step 6). Upon receiving the collaboration requirement graph, the collaborating party calculates its own collaboration capability graph (Step 7) and, in conjunction with the requester's collaboration requirement graph, calculates the collaboration graph response (Step 8). The requester calculates the collaboration ordering graph based on the collaborating party's collaboration response (Steps 9 and 10), and sets the size of the collaboration ordering graph in the feature segmentation module based on the collaboration graph (Step 11). Based on the set collaboration ordering graph, the ordering collaboration feature data can be obtained, i.e., which dimensions of feature data the requester needs the collaborating party to send. The ordering collaboration feature data is then sent to the collaborating party (Step 12).
[0049] At time step t+1, after receiving the ordered collaboration feature data, the collaborating party performs feature segmentation based on the collaboration graph (step 13) and sends the collaboration feature data back to the demand party (step 14). Importantly, at this point, the demand party has already obtained the new frame of perception data (steps 18 and 19) and performed feature extraction (steps 20 and 21). When the collaboration feature data arrives, the demand party performs feature scaling on it (step 16) to obtain sparse features (step 17). These sparse features are then fused with the features from the current time step (t+1) using multi-source feature fusion (step 22). This process is continuously iterated. In each new time step, the demand party calculates a new collaboration capability graph (step 24) and updates it (step 25) to ensure the continuous progress of the collaboration process.
[0050] like Figure 3 As shown, from the perspective of the collaborating party, the collaboration process is as follows: At time step t, the collaborating party first processes the raw sensing data through the sensing processing module, such as... Figure 3 Step 1 involves sending the preprocessed data to the feature extraction module (Step 2). The feature extraction model extracts features and sends them to the collaboration graph generation and management module (Steps 3 and 4). The collaboration graph generation and management module calculates the collaboration capability graph (Step 5) and publishes it to the collaboration control module based on the collaboration graph (Step 6). When the requester sends a collaboration requirement graph (Steps 7 to 9), the collaboration control module of the collaborator combines the requester's collaboration requirement graph with its own collaboration capability graph to calculate the collaboration graph response (Step 10) and sends it to the requester (Step 11). The requester then calculates the collaboration order graph based on its own needs and sends it to the collaborator (Steps 12 and 13).
[0051] At time step t+1, after the collaborating party receives the order collaboration feature data information and collaboration order graph from the demand party, as shown in steps 14-15, a new round of collaboration feature data processing is initiated. At this time, the perception processing module has already started processing the perception data for the next time step, as shown in steps 16-17, and the feature extraction module is also processing new features, as shown in steps 18-19. The feature segmentation module based on the collaboration graph will segment these new feature data, as shown in steps 20-21, and finally send the segmented feature data to the demand party through the collaboration control module, as shown in step 22. The demand party performs feature scaling and multi-source feature fusion based on the collaboration graph, as shown in steps 23-24.
[0052] Within each time step, both the demander and the collaborator repeat the collaborative control process described above in the current time step t, as well as the process of transmitting collaborative features and fusing multi-source features based on the order collaborative feature data from the previous time step.
[0053] like Figure 4 and Figure 5 The diagram illustrates the process of intelligent agent collaborative control under a passive acceptance by the demand side. Figure 4 and Figure 5 This paper illustrates the collaboration process from both the demander's and collaborators' perspectives in a passive demander-acceptance strategy. In the passive demander-acceptance strategy, collaborators proactively publish a collaboration capability map to the demander. The demander passively calculates collaboration requirements based on this map and orders feature data from the collaborators. The collaborators then send the feature data back to the demander for sparse feature extraction and fusion. The demander updates the collaboration capability map at each time step, passively maintaining the continuous collaboration process.
[0054] like Figure 4 As shown, from the perspective of the demand side, the collaboration process is as follows: At time step t, the collaborating party first calculates the collaboration capability map using its own perceived data, as shown below. Figure 4 Step 1 involves publishing the collaborative capability diagram to the requester (Step 2). The requester then receives the collaborative capability diagram published by the collaborator (Step 3). The requester calculates the collaborative capability diagram based on its current self-perceived data and publishes it to the collaborative control module based on the collaborative diagram (Steps 4-7). The collaborative control module based on the collaborative diagram calculates the collaborative requirement diagram (Step 8) and, based on the collaborator's collaborative capability diagram, calculates and sets the collaborative ordering diagram (Steps 9-10).
[0055] At time step t+1, the demander orders feature data from the collaborator based on the collaborative ordering graph. The collaborator performs feature segmentation based on the collaborative graph and sends the collaborative feature data to the demander, as shown in steps 11-13. The demander extracts features based on the perceived data at the current time step, as shown in steps 17-20. The feature segmentation module based on the collaborative graph receives the collaborative feature data, as shown in steps 14-15, and then combines the sparse features obtained by scaling the collaborative feature data with the features extracted at the current time step, as shown in step 21. This process will continuously loop. In each new time step, the demander will calculate a new collaborative capability graph, as shown in step 23, and update it, as shown in step 24, to ensure the continuous progress of the collaboration process.
[0056] like Figure 5 As shown, from the perspective of the collaborating party, the collaboration process is as follows: At time step t, the collaborating party first processes the raw sensing data through the sensing processing module, such as... Figure 5Step 1: The processed data is sent to the collaboration graph generation and management module (Step 2), where a collaboration capability graph is calculated (Step 3). Subsequently, the collaboration capability graph is published to the collaboration control module based on the collaboration graph (Step 4). Upon receiving the collaboration capability graph, the collaboration control module actively shares it with the requester (Step 5). The requester calculates the collaboration capability graph and collaboration requirement graph using its own perceived data (Steps 6-7), then calculates the collaboration ordering graph based on the shared collaboration capability graph (Step 8), and determines the ordering collaboration feature data to send to the collaborator (Step 9). The collaborator initiates the collaboration feature data sending process (Step 10), saving the collaboration ordering graph and ordering collaboration feature data to the feature segmentation module based on the collaboration graph (Step 11).
[0057] At time step t+1, the collaborator extracts features based on the perception data of the current time step as in steps 12-15, and segments the features according to the ordered collaborative feature data as in step 16. The segmented feature data is then sent to the demander as in steps 17-18. The demander performs feature scaling and multi-source feature fusion based on the collaboration graph as in steps 19-20.
[0058] Within each time step, both the demander and the collaborator repeat the collaborative control process described above in the current time step t, as well as the process of transmitting collaborative features and fusing multi-source features based on the order collaborative feature data from the previous time step.
[0059] The system of this invention can flexibly select collaborative strategies according to scenario requirements, thereby optimizing information transmission, enhancing perception quality, and improving decision-making accuracy under different perception conditions and task requirements. To overcome the performance limitations of sensors that can be carried by a single intelligent machine and to compensate for the limited communication capabilities between intelligent machines, this invention proposes a collaboration graph as a tool for abstractly representing the collaborative relationships and data interaction between intelligent machines. The collaboration graph can flexibly express information such as needs, capabilities, and feedback, and by dynamically updating the graph's information, it makes the collaboration of intelligent machine groups more flexible and efficient.
[0060] In this invention, the collaboration graph serves as the core tool. Through the interaction of the collaboration requirement graph, collaboration capability graph, and collaboration ordering graph, it accurately describes the perception needs of the demanding intelligent machine and the capability feedback of the collaborating intelligent machine. The collaborative perception between intelligent machines is essentially the realization of blind-spot collaborative perception based on the collaboration graph, which corresponds to the collaborative graph-based intelligent machine group collaboration method proposed in this invention.
[0061] This invention utilizes a collaborative graph-based intelligent communication and data exchange mechanism to achieve comprehensive perception and efficient processing of blind spots. First, it extracts and identifies blind spots from raw point cloud data from sensors of intelligent machines such as autonomous vehicles and drones, abstracting them into a collaborative demand map. Simultaneously, a collaborative ordering map is calculated based on the collaborative capability maps of collaborating intelligent machines (such as other intelligent vehicles, drones, and roadside equipment) and its own collaborative demand map. Collaborators then perform feature segmentation and data exchange based on the collaborative ordering map, ultimately achieving feature fusion. Finally, the intelligent machine makes a series of decisions based on the fused features. The collaborative capability map, obtained by identifying and smoothing dense areas from point cloud data, represents the effective perception area of the intelligent machine. The collaborative demand map, on the other hand, represents the sparse portion of the point cloud data, indicating the machine's perception blind spots. The collaborative ordering map is obtained by matching the demander's collaborative demand map with the collaborator's collaborative capability map. This collaborative ordering map describes the data area and feature scale requested by the demander from the collaborator.
[0062] Taking vehicle-road cooperation as an example, one implementation of the intelligent machine swarm cooperation method based on cooperation graph of the present invention is as follows: Figure 6 As shown, in a vehicle-road cooperative perception scenario, the autonomous vehicle first uses its own sensor data (such as LiDAR, cameras, etc.) to perceive the surrounding environment, but there may be multiple perception blind spots. To solve this problem, the vehicle identifies a cooperative capability map based on its own perception data, and then obtains a cooperative demand map corresponding to the cooperative capability map. The cooperative demand map represents perception blind spots or sparse areas. The vehicle transmits this information, along with the cooperative demand map and location data, to the intelligent roadside unit (the collaborating party) via wireless communication. The intelligent roadside unit uses its deployed high-precision sensors (such as cameras, radar, LiDAR, etc.) to collect data about the surrounding environment and create a cooperative capability map. The collaborating party calculates a cooperative ordering map based on its own cooperative capability map and the demanding party. After confirmation by the vehicle, the collaborating party orders the collaborating party's feature data. The collaborating party performs feature segmentation and returns sparse features. The vehicle receives the sparse features and performs scaling and attention-based multi-source feature fusion. After obtaining the fused features, the vehicle performs decision processing. When roadside units actively share data, they not only push environmental data to nearby autonomous vehicles in real time, but also optimize the relevance and accuracy of this data through collaborative graphs. This helps vehicles more comprehensively cover and compensate for their blind spots, thereby improving driving safety and efficiency. Conversely, when vehicles share their blind spot data with roadside units, the roadside units integrate and analyze this information, feeding back more accurate blind spot compensation data to the vehicles, optimizing their perception range and decision-making capabilities. This two-way dynamic data exchange and collaborative working mode significantly improves the accuracy and efficiency of blind spot perception, supporting the safe and smooth operation of intelligent transportation systems.
[0063] like Figure 6 As shown below, the three parts of perception processing, collaborative control, and collaborative cognition are described respectively.
[0064] (1) Perception Processing. Perception processing refers to the process by which intelligent machines receive raw data from hardware sensing devices such as cameras and lidar, and perform preliminary filtering and synchronization to ensure the timeliness and accuracy of the data. In this embodiment of the invention, the perception processing module mainly processes point cloud data from the main vehicle and roadside equipment. The specific processing steps include data reception, preprocessing, filtering, and synchronization.
[0065] Data reception: The main vehicle and roadside equipment collect point cloud data through their respective sensing devices, such as lidar.
[0066] Data preprocessing: The received point cloud data is voxelized to reduce the data volume and improve processing efficiency. In this embodiment of the invention, a three-dimensional mesh coordinate system is constructed with the vehicle's centroid as the origin. The X-axis is aligned with the vehicle's driving direction, the Z-axis is perpendicular to the ground, and the Y-axis is perpendicular to the vehicle's driving direction. Each axis of the mesh is set according to a predetermined voxel size d. x d y d z Perform equal-interval partitioning. The specific voxelization definition is as follows:
[0067]
[0068] Where, p x p y p z These are the coordinates of point p on the three axes. V(x,y,z) represents the set of points in space, where V(x,y,z) are the voxelized coordinates of point p.
[0069] Data filtering and synchronization: The voxelized point cloud data is filtered to remove noise and irrelevant points, and synchronized with the time step to ensure that the data from different sensing devices are aligned within the same time frame.
[0070] In the application scenarios of this invention, intelligent devices can preprocess the acquired point cloud data in the manner described above. For data acquired by other sensing devices, such as images, preprocessing can be performed according to pre-defined requirements such as size and resolution. After preprocessing the data acquired by different sensors, noise reduction and time synchronization are performed.
[0071] (2) Collaborative control. This invention addresses the blind spots of intelligent machines by using a collaborative graph to achieve feature collaboration and interaction between intelligent machines. Therefore, the collaborative control module includes three parts: blind spot recognition, collaborative graph interaction, and feature segmentation.
[0072] (2.1) Blind Spot Recognition. The main objective of blind spot recognition is to identify the perception blind spots from the vehicle's own perception data. To this end, this invention employs a blind spot recognition method based on point cloud data. Judging blind spots based on point cloud density is an effective method because point cloud density represents the amount of information and reflects the effective information content of the area.
[0073] To assess the point cloud density of each region, this invention employs a 3D-to-2D projection method to calculate the point density ρ(x, y) of the 2D voxels as follows:
[0074]
[0075] Where count(V(x, y, :)) represents the number of points contained in voxel V(x, y, :) on the (x, y) plane. Based on a preset density threshold θ, the point density ρ(x, y) is binarized to generate a blind zone mask M, thus obtaining the collaborative requirement map, where:
[0076]
[0077] In this model, M(x, y) = 1 indicates that the region is a blind zone for the intelligent machine, and M(x, y) = 0 indicates that the region is a visible region for the intelligent machine. To eliminate noise and small-area errors, morphological operations such as erosion and dilation are performed on the generated mask M to optimize its quality. Subsequently, connected component analysis is applied to identify individual blind zone regions. This blind zone information will be used in subsequent collaborative sensing processes. The result of blind zone identification is shown below. Figure 7 As shown, the location and shape of the blind spots perceived by the main vehicle are illustrated, with sparsely gridded areas representing the blind spots perceived by the vehicle.
[0078] (2.2) Collaboration Graph Interaction. Collaboration graph interaction is the core part of the collaborative control module, responsible for generating, sharing, and processing collaboration graphs between the main vehicles and roadside equipment to achieve effective data transmission and feature sharing. This process includes the generation of the collaboration demand graph, the sharing of the collaboration capability graph, and the calculation and transmission of the collaboration order graph. The specific process and formula are described below:
[0079] Collaborative Demand Map Generation: The main vehicle, i.e., the demand-side intelligent machine, generates a collaborative demand map based on its own perception data. The primary purpose is to identify its perception blind spots, requiring roadside equipment, i.e., the collaborating intelligent machines, to provide perception features of the relevant areas. The generation process of the collaborative demand map is consistent with the blind spot mask generation process, defined as follows:
[0080] R = M;
[0081] Where R represents the collaborative requirement map, and M is the blind spot mask generated by the main vehicle. The collaborative requirement map R is used to represent the blind spot areas that exist in the environmental perception of the demand party, and the collaborating party needs to provide the perception features of these areas.
[0082] Collaborative Capability Map Generation and Sharing: Roadside equipment uses its own sensing data to generate a collaborative capability map, representing the sensing feature areas it can provide. The generation process of the collaborative capability map is similar to that of the collaborative demand map, and the specific steps are as follows:
[0083] a) Perception Data Preprocessing: Based on the relative pose between the collaborating party and the demand party, the collaborating party's raw point cloud data is transformed to the demand party's coordinate system using a coordinate transformation matrix to ensure spatial alignment of the perception data of both parties. In the demand party's coordinate system, the transformed point cloud is projected from 3D to 2D, and the 2D voxel density is calculated according to the same voxel partitioning rules as for blind zone identification.
[0084]
[0085] Where, count(V) collab (x,y,:) represents the voxel V in the (x,y) plane. collab The number of points contained in (x,y,:). The density is compared with a threshold θ1 to generate a collaborative capability map C, as follows:
[0086]
[0087] ρ here collab (x,y)≥θ1 indicates that the point cloud density ρ of the collaborating party in this region is... collab (x,y) is sufficiently high to provide reliable sensing capabilities. In the method of this invention, thresholds θ1 and θ can be set according to the actual application. If the thresholds are set to be equal, the collaborative capability map and collaborative demand map of the intelligent machine within the same time step are complementary.
[0088] b) Collaborative Capability Map Sharing: Roadside equipment will share the collaborative capability map C collab The data is transmitted over the network to the main vehicle, enabling it to understand which areas the roadside equipment can provide sensing characteristics for.
[0089] In the demand-side proactive request mode, the intelligent machine, acting as a collaborator, calculates a collaboration graph response after receiving the collaboration requirement graph sent by the demander. The intelligent machine first converts the received collaboration requirement graph into its own coordinate system, then calculates its intersection with its own collaboration capability graph. If the intersection is not empty, the collaboration graph response is fed back to the demander. The demander calculates a collaboration ordering graph after receiving the collaboration graph response. In this embodiment of the invention, the demander calculates the collaboration ordering graph based on all responses and receives supplementary data from all collaborators.
[0090] In the demand-side passive acceptance mode, after obtaining the collaborative capability map from the collaborator, the demander transforms it to its own coordinate system and then calculates the intersection of the map with its own collaborative demand map to calculate the collaborative ordering map. In this embodiment of the invention, the calculation process of the collaborative ordering map in this mode is as follows: the main vehicle receives the collaborative capability map shared by the roadside equipment and performs coordinate system transformation to obtain collaborative capability map C. collab Based on the self-generated collaboration requirement graph R, the collaboration ordering graph S is calculated. The collaboration ordering graph represents the effective feature area that the collaborating party can transmit for the main vehicle, i.e., the collaborating party's collaboration capability graph C. collab The intersection of the collaboration requirement diagram R with the demand side, S = C collab ∩R.
[0091] The calculation diagram for collaborative ordering in this embodiment of the invention is as follows:
[0092] Where S(x,y)=1 indicates that at location (x,y), the roadside equipment can provide the sensing features required by the demander, and S(x,y)=0 indicates that the transmission conditions are not met at this location.
[0093] Collaborative ordering map transmission: The main vehicle will send the calculated collaborative ordering map S to the loop-side equipment, indicating which areas' feature data need to be transmitted.
[0094] (2.3) Blind Spot Feature Segmentation. The main vehicle selects the corresponding blind spot features, i.e., the cooperative feature data, based on the cooperative ordering map S, to reduce data transmission volume and improve communication efficiency. After receiving the cooperative feature data sent by the main vehicle, the roadside equipment performs feature segmentation and sends the segmented features to the main vehicle.
[0095] Since the spatial dimensions of the feature map and the collaborative ordering map may differ, it is necessary to perform upsampling or downsampling operations on the collaborative ordering map S to match the size of the feature map. The feature extraction module of this invention extracts multi-scale features from the perceptual data, performs upsampling or downsampling operations on the collaborative ordering map according to the spatial dimensions of each scale feature, so that the collaborative ordering map matches the size of the feature at that scale, and determines the ordering collaboration feature data at that scale based on the matched collaborative ordering map Upsample(S).
[0096] After obtaining Upsample(S), the roadside equipment uses a feature extraction module to obtain the current multi-scale features F. road For each scale of features Upsample(S) is applied to the feature map. The specific steps are as follows:
[0097]
[0098] Upsample(S) adjusts S to match... For the same spatial dimensions, ⊙ represents element-wise multiplication. This way, only features at the blind zone locations are preserved, while features at other locations are set to zero. Feature map after masking. This represents the segmentation feature at the k-th scale.
[0099] Since only the features corresponding to the blind zone need to be transmitted, the feature map Due to its sparsity, the roadside equipment will further compress and encode the segmented multi-degree features to reduce the amount of data transmitted. After receiving the collaborative features from the collaborating party, the main vehicle will scale and convert them into sparse features.
[0100] (3) Collaborative cognition. Collaborative cognition includes feature extraction and feature fusion. In this method, it refers to extracting point cloud feature data and fusing multi-source features to obtain enhanced features for subsequent decision-making.
[0101] (3.1) Feature Extraction. Feature extraction involves extracting multi-scale features from perceived data to provide rich environmental perception information for the main vehicle. First, the PointPillars method is used to process the acquired point cloud data. The PointPillars method converts the point cloud data into a regular 3D mesh by voxelizing it, and then converts the data into a 2D pseudo-image format. This conversion preserves height information and makes the data suitable for a 2D convolutional neural network architecture, improving computational efficiency. Next, a feature extraction network G is used. θ The (·) network, the backbone of PointPillars, performs in-depth feature extraction on voxelized point cloud data. This network includes a series of convolutional layers and activation functions, effectively extracting spatial and semantic information to generate basic feature maps.
[0102] To obtain multi-scale feature representations, this invention employs a multi-scale feature extraction module (FPN). θ (·), this module utilizes a top-down structure to gradually fuse features from high to low levels, achieving the integration of multi-scale information. For example... Figure 8As shown, this embodiment of the invention implements a 3-layer multi-scale feature extraction network. By establishing lateral connections between feature maps of different scales, high-level semantic features can be transferred to lower layers while preserving high-resolution detail information of low-level features. This structure can be flexibly extended to more scales to adapt to the needs of different tasks. In the multi-scale feature extraction process, the high-level feature maps are first upsampled using deconvolution or bilinear interpolation to match the spatial size of the low-level feature maps. Subsequently, the number of channels in the low-level feature maps is adjusted using 1×1 convolution, which not only unifies the channel dimension but also helps reduce noise. Next, feature fusion is achieved by adding the upsampled high-level feature maps to the adjusted low-level feature maps element-wise. Finally, 3×3 convolution is applied to these fused feature maps for further feature refinement and noise suppression, thereby enhancing the expressive power and information richness of the features. After the above processing, a multi-scale feature set F is generated. ego as follows:
[0103]
[0104] in, This represents the perception data of the main vehicle, where K is the number of feature layers. This represents the k-th feature layer.
[0105] (3.2) Feature fusion. Feature fusion refers to the fusion of the vehicle's own perception features with blind spot features from roadside equipment to improve the overall perception of the environment.
[0106] a) Attention-based feature fusion. To effectively fuse multi-scale features, an attention-based fusion method was adopted. This method is used for the features of the main vehicle. and received blind zone features Perform linear mappings to obtain query, key, and value vectors respectively:
[0107]
[0108] Among them, W Q W K W V It is a trainable weight matrix.
[0109] Calculate attention score
[0110]
[0111] Where, d k It represents the dimension of the feature, used for scaling to prevent the value from becoming too large. The superscript T indicates transpose.
[0112] The value vector is weighted using attention scores:
[0113]
[0114] The features of the main vehicle can be fused with the weighted roadside features using either addition or concat operations:
[0115]
[0116] or
[0117]
[0118] Finally, the fusion features are obtained.
[0119] b) Fusion of multi-scale features. To unify the size of the feature maps and facilitate subsequent processing, the fused multi-scale features need to be adjusted. For each fused feature map... Upsampling is performed using the decoder network to ensure consistent spatial dimensions:
[0120]
[0121] The decoder can consist of a series of deconvolutional or upsampling layers, effectively restoring the spatial resolution of the feature maps. Subsequently, feature maps from all scales are concatenated along the channel dimension to form a comprehensive feature map F. ego,final as follows:
[0122]
[0123] This integrated feature map contains information at different scales, possesses stronger expressive power, and can effectively improve the performance of subsequent perception tasks. Specifically, the integrated feature map F after fusion perception... ego,final It can be used as input for downstream tasks, such as connecting to an object detection head for object detection and localization in a scene, or connecting to a semantic segmentation head for pixel-level scene understanding. Furthermore, this feature map can also be used for other perception tasks, such as drivable area prediction and depth estimation; the specific application depends on the task requirements and the network architecture design.
[0124] To demonstrate the effectiveness of the present invention's technical solution in improving perception performance and reducing communication bandwidth consumption, a large-scale cooperative perception dataset was collected using the Carla simulation system, and experiments were conducted on machines with NVIDIA RTX 3090 GPUs and 12th generation Intel(R) Core(TM) i9-12900K CPUs. As shown in Table 1, multi-vehicle perception significantly outperforms single-vehicle perception in terms of AP@IoU = 0.5. In multi-vehicle perception, compared to the basic multi-vehicle perception method (sharing all perception features), the cooperative graph-based method of the present invention reduces the average communication cost by 29.5% at the cost of a 1.8% performance decrease. This demonstrates the advantages of the present invention's technical solution in communication-constrained scenarios.
[0125] Table 1 Performance comparison between the present invention and existing methods
[0126] method Average traffic AP@0.5 Single-vehicle sensing - 62.4 Multi-vehicle perception 4MB 74.3 Multi-vehicle perception - based on collaboration graph 2.82MB 72.9
[0127] It should be noted that this embodiment can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software executed by a controller, microprocessor, or other computing device. When aspects of this embodiment are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatus, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.
[0128] Except for the technical features described in the specification, all other technologies are known to those skilled in the art. Descriptions of well-known components and technologies are omitted in this invention to avoid redundancy and unnecessary limitation. The embodiments described above do not represent all embodiments consistent with this application. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this invention are still within the protection scope of this invention.
Claims
1. A collaborative system for intelligent machine swarms based on a collaboration graph, characterized in that, Deploy a perception processing module, a collaborative control module, and a collaborative cognition module on each intelligent machine participating in the collaboration; The perception processing module acquires raw perception data from various perception devices, performs voxelization preprocessing on the point cloud data collected by the intelligent machine, and cleans and synchronizes the acquired perception data over time. The collaborative control module includes a collaborative graph generation and management module, a collaborative graph-based collaborative control module, and a collaborative graph-based feature segmentation module. The collaboration map generation and management module receives the perception data transmitted by the perception processing module in real time, and generates and updates the collaboration capability map. The collaboration capability map is obtained by: projecting the voxelized point cloud data perceived by the intelligent machine at the current moment from three dimensions to two dimensions, calculating the two-dimensional voxel density. The higher the density, the stronger the perception capability of the intelligent machine in the corresponding area. The two-dimensional voxel density is compared with the preset density threshold A. If it is less than the density threshold A, the value of the corresponding area plane is set to 0; otherwise, it is set to 1, thus obtaining the collaboration capability map. The collaborative control module based on the collaborative graph calculates the collaborative demand graph based on the current perception data of the intelligent machine, and performs collaborative control between intelligent machines according to the set collaborative control mode. The collaborative ordering graph is obtained by finding the intersection of the collaborative demand graph of the demander and the collaborative capability graph of the collaborator. The collaborative demand graph is obtained by: projecting the voxelized point cloud data perceived by the intelligent machine at the current moment from three dimensions to two dimensions, calculating the two-dimensional voxel density, comparing the two-dimensional voxel density with a preset density threshold B. If it is less than the density threshold B, the value of the corresponding region plane is set to 1; otherwise, it is set to 0, thus obtaining the collaborative demand graph. The collaborative control mode includes two types: one is the demand-side active request mode, in which the intelligent machine acting as the demand side actively broadcasts its own collaborative demand map to seek collaboration with intelligent machines with corresponding perception capabilities; the other is the demand-side passive acceptance mode, in which the intelligent machine acting as the collaborator actively broadcasts its own collaborative capability map and coverage area to other intelligent machines for them to determine whether collaboration is needed. The feature segmentation module based on the collaboration graph is implemented as follows: On the one hand, the demand side scales the collaboration ordering graph according to the spatial size of the features extracted from the perception data to determine the ordering collaboration feature data. This data records which dimensions of collaboration features the collaborator needs to send. On the other hand, the collaborator segments the features extracted at the current time step according to the received ordering collaboration feature data, and sends the segmented features to the demand side through the collaboration control module based on the collaboration graph. The collaboration control module upsamples or downsamples the collaboration ordering graph according to the spatial size of each scale feature extracted by the demand side from the perception data, so that the size of the collaboration ordering graph matches the size of the scale feature. The matched collaboration ordering graph is multiplied element-wise with the scale feature to obtain the ordering collaboration feature data at that scale. The collaborative cognition module is equipped with a feature extraction module and a feature fusion module. The feature extraction module extracts features from its own perception data of each frame, and at the same time scales and converts the collaborative features of the collaborating party into sparse features. The feature fusion module fuses the features extracted at the current time step with the collaborative features, and the demand side uses the fused features to make decisions. Specifically, the demand side scales the collaborative features into sparse features, and then calculates the weights of the sparse features based on the attention mechanism at each scale. The weighted sparse features are then added or concatenated with the current features to obtain the fused features at each scale. The spatial dimensions of the features at each scale are then adjusted to be consistent, and the feature maps of all scales are stitched together along the channel dimension to form a comprehensive feature map.
2. The system according to claim 1, characterized in that, The intelligent machine is configured in a demand-driven request mode. In this mode, the collaborative control process between the demander and the collaborator includes: Within time step t, the demander calculates the collaboration requirement graph based on its current perception data and publishes the collaboration requirement graph; the collaborator calculates the collaboration capability graph based on its current perception data, calculates the collaboration graph response after receiving the demander's collaboration requirement graph, and sends it back to the demander; the demander calculates the collaboration order graph based on the collaborator's feedback collaboration graph response, determines the order collaboration feature data, and sends it to the collaborator. Within time step t+1, after receiving the order collaboration feature data sent by the demand party, the collaborating party starts collaboration feature data processing, extracts features from the perception data of time step t+1, segments the features according to the order collaboration feature data, and sends the segmented collaboration features to the demand party; the demand party scales the received collaboration features and merges them with the features it extracted within time step t+1. Within each time step, both the demander and the collaborator repeat the collaborative control process described above.
3. The system according to claim 1, characterized in that, The intelligent machine is configured in a demand-side passive acceptance mode. In this mode, the collaborative control process between the demand-side and the collaborating party includes: Within time step t, the collaborating party calculates the collaboration capability map based on its current perception data and publishes it; the demanding party calculates the collaboration demand map based on its current perception data, and after obtaining the collaborating party's capability collaboration map, calculates the collaboration ordering map, determines the ordering collaboration feature data, and sends it to the collaborating party. Within time step t+1, the collaborating party extracts features based on its current self-perceived data, segments features according to the order collaboration feature data of the demand party, and sends the segmented collaboration features to the demand party; the demand party extracts features from its current self-perceived data, scales the received collaboration features, and merges all received collaboration features with its own features. Within each time step, both the demander and the collaborator repeat the collaborative control process described above.
4. The system according to claim 1 or 2, characterized in that, The collaboration graph response is obtained by the collaborating party transforming the received collaboration requirement graph into its own coordinate system, then finding the intersection with the collaborating party's collaboration capability graph. If the collaboration graph response is not empty, the collaborating party sends the collaboration graph response back to the requesting party; otherwise, it does not send a response.
5. A collaborative method for intelligent machine swarms based on collaboration graphs, characterized in that, Includes the following steps: Step 1: Deploy a perception processing module on the intelligent machine to process the data collected by the machine's perception devices. This includes voxelization preprocessing of the collected point cloud data and cleaning and time synchronization processing of the acquired perception data. Step 2: Set up a collaborative control module on each intelligent machine. The collaborative control module identifies its own blind spots, sets up the collaborative graph interaction mode, and performs feature segmentation. The collaborative control module calculates the point density of two-dimensional voxels and compares it with a set threshold to generate a blind zone mask. The blind zone mask describes the blind zone area that exists in the intelligent machine's environmental perception. The collaborative control module obtains the collaborative requirement map of the intelligent machine itself from the blind zone mask. The collaborative requirement map is obtained by: projecting the voxelized point cloud data perceived by the intelligent machine at the current moment from three dimensions to two dimensions, calculating the two-dimensional voxel density, comparing the two-dimensional voxel density with a preset density threshold B. If it is less than the density threshold B, the value of the corresponding region plane is set to 1; otherwise, it is set to 0, thus obtaining the collaborative requirement map. The collaborative control module calculates the collaborative capability map by projecting the voxelized point cloud data currently perceived by the intelligent machine from three dimensions to two dimensions, calculating the two-dimensional voxel density. The higher the density, the stronger the perception capability of the intelligent machine in the corresponding area. The two-dimensional voxel density is compared with a preset density threshold A. If it is less than the density threshold A, the value of the corresponding area plane is set to 0; otherwise, it is set to 1, thus obtaining the collaborative capability map. The collaborative control module obtains the collaborative ordering diagram by finding the intersection of the collaborative requirements diagram of the demand side and the collaborative capability diagram of the collaborators. The collaborative control module's collaborative graph interaction mode refers to the implementation of collaborative control between the demand side and the collaborating side based on the collaborative demand graph, collaborative capability graph, and collaborative ordering graph; there are two collaborative control modes: One is the demand-side proactive request mode, in which the intelligent machine acting as the demand side actively broadcasts its own collaboration demand map to seek collaboration with intelligent machines that have the corresponding perception capabilities; the other is the demand-side passive acceptance mode, in which the intelligent machine acting as the collaborator actively broadcasts its own collaboration capability map and coverage area to other intelligent machines to determine whether collaboration is needed. Feature segmentation in the collaborative control module refers to the demander determining which dimensions of collaborative features to order from the collaborator based on the collaborative ordering diagram. The collaborative control module extracts multi-scale features from the perceptual data by the demander and performs upsampling or downsampling operations on the collaborative ordering diagram according to the spatial size of each scale feature, so that the collaborative ordering diagram matches the size of the scale feature. Based on the matched collaborative ordering diagram, the ordering collaborative feature data at that scale is determined. The matched collaborative ordering graph is multiplied element by element with the feature at that scale to obtain the collaborative ordering feature data at that scale. Step 3: The intelligent machine extracts multi-scale features from the current sensing data and performs feature fusion after receiving collaborative features sent by the collaborator as the demand side. At the current time step, both the demand side and the collaborating side extract multi-scale features based on their own perceived data; Then, the collaborating party segments its own multi-scale features based on the order collaboration feature data sent by the demand party in the previous time step, and transmits the segmented multi-scale features as collaboration features to the demand party. The demand party scales the collaboration features into sparse features, and then calculates the weights of the sparse features based on the attention mechanism at each scale. The weighted sparse features are added or concatenated with the current features to obtain the fused features at each scale. Then, the spatial size of the features at each scale is adjusted to be consistent, and the feature maps of all scales are stitched together in the channel dimension to form a comprehensive feature map.
Citation Information
Patent Citations
Vehicle control system, roadside equipment and vehicle-road cooperation system
CN113724519A
Multi-vehicle cooperative environment perception method based on semantic-level information fusion
CN114091598A
Multi-vehicle cooperative sensing method and device based on laser radar and storage medium
CN116893424A
Vehicle-road multi-mode data instance-level feature transmission collaborative awareness system and method and storage medium
CN118741460A
Multi-device cooperation method, device and system and medium
CN112235733A
