A heterogeneous multi-robot cooperative perception and denial system and method

Abstract

This invention belongs to the field of security, specifically relating to a heterogeneous multi-robot collaborative perception and denial system and method. The method includes: heterogeneous robot nodes preprocessing multimodal environmental data and extracting semantic features; a central platform performing cross-modal fusion to construct a global situation map and quantifying performance indicators into capability vectors; extracting risk features from the situation map for level assessment and generating task requirements; calculating a scheduling matching score by matching the capability vectors with the task requirements; the scheduled robot calculating an intent score based on the target's kinematic behavior and posture features, fusing local risk scores, and initiating graded denial accordingly, jointly triggering higher-level denial measures with the global risk assessment value; and the central platform updating the fusion model parameters and using reinforcement learning to optimize the scheduling strategy. This invention significantly reduces bandwidth and computational load, improves situational awareness accuracy, achieves efficient heterogeneous robot collaboration and accurate graded denial, and possesses continuous self-optimization capabilities.

Description

Technical Field

[0001] This invention belongs to the field of security, specifically relating to a heterogeneous multi-robot collaborative perception and denial system and method. Background Technology

[0002] Current industry solutions suffer from several drawbacks. The first is the traditional centralized, single-modal, independent perception system. These systems require robots to directly upload raw data to a central location, where each modality's data is processed separately before being stitched together to form a situational awareness map. This not only consumes significant bandwidth for raw data transmission but also lacks underlying alignment and fusion of cross-modal data, resulting in fragmented and low-precision global situational information, and placing a heavy burden on the central processing unit, leading to latency. This application addresses this by using edge-side preprocessing to reduce dimensionality and transmit only feature vectors, drastically reducing bandwidth pressure. Simultaneously, the attention mechanism enables cross-modal fusion, resulting in a more accurate and comprehensive global situational awareness. The second is a heterogeneous robot scheduling system based on fixed rules. These systems pre-define fixed task allocation logic and cannot dynamically adjust based on real-time situational awareness and robot performance, resulting in low scheduling matching and potential resource waste or response delays. This application achieves precise matching between tasks and robot capabilities by obtaining standardized capability vectors through hash mapping and combining this with dynamic matching and scoring scheduling. The third is a non-lethal rejection system triggered by a single threshold. These systems trigger rejection based on only a single indicator, failing to accurately assess the target's potential intent, resulting in a high false positive rate and a one-size-fits-all rejection approach. This application achieves graded rejection through composite intent scores and joint risk assessment, reducing false positives and precisely controlling rejection strength. The fourth type is a static collaborative system without closed-loop optimization. These systems have fixed models and strategies, cannot adapt to environmental changes, and their performance declines over time. This application achieves closed-loop iteration through incremental learning and reinforcement learning, continuously improving system performance and enhancing adaptability. Summary of the Invention

[0003] The technical problem this invention aims to solve is to provide a heterogeneous multi-robot collaborative perception and rejection system and method. Through a semi-decentralized architecture combining edge-end multimodal perception compression and center-end cross-modal deep fusion, and integrating heterogeneous robot capability structure quantification, target intent composite evaluation, and hierarchical rejection mechanisms, and using incremental learning and reinforcement learning to form a closed-loop iterative optimization, a multi-robot collaborative perception and rejection system with low bandwidth consumption, high-precision situational awareness, efficient collaborative scheduling, and accurate non-lethal rejection is achieved.

[0004] This invention provides a heterogeneous multi-robot cooperative perception and rejection method, comprising the following steps: S1: Multiple heterogeneous robot nodes synchronously collect environmental data through their onboard multimodal sensors, preprocess and extract features from the collected data locally, generate compressed feature vectors representing the semantics of the environment, and upload them to the central platform. S2, the central platform receives feature vectors and raw data uploaded by each heterogeneous robot node, performs cross-modal data association and fusion, generates unified multimodal fusion features and constructs a global situation map covering the monitoring area based on these features; at the same time, based on the identity of each robot, it obtains its capability indicators and quantifies them into structured capability vectors; S3, the central platform extracts high-level risk features based on the global situation map, performs risk level assessment and generates task requirements; calculates the matching degree between the capability vector of each robot and the task requirements, and performs attenuation correction based on the spatial distance of the robot's current position relative to the task area to obtain the scheduling matching score, and dynamically allocates tasks with the goal of optimizing the global response time. S4, the scheduled heterogeneous robot nodes continuously track the designated target, calculate the intent score based on the target's kinematic behavior and posture characteristics, and generate a local risk score by combining the behavior anomaly degree; initiate a hierarchical rejection response based on the local risk score, and initiate the corresponding level of rejection measures when the local risk score and the global risk assessment value issued by the central platform reach the preset conditions. S5, the central platform continuously collects feedback data on perception accuracy and rejection effectiveness, updates the parameters of the cross-modal fusion model incrementally, and continuously optimizes the task allocation strategy through reinforcement learning based on the experience tuple consisting of situational state, scheduling actions and effectiveness rewards, forming an iterative closed loop of perception, decision-making, execution and optimization.

[0005] Furthermore, in step S1, the synchronous acquisition of environmental data includes: providing a local time reference through a hardware clock, correcting deviations using a network time protocol, and achieving microsecond-level timestamp alignment between the data streams of each modal sensor; The preprocessing includes: applying Gaussian filtering to the visible light image, applying median filtering to the infrared data, removing statistical outliers from the lidar point cloud based on Mahalanobis distance, and normalizing the data for each modality. The feature extraction is achieved through a lightweight convolutional neural network deployed locally on the robot.

[0006] Furthermore, in step S2, the cross-modal data association and fusion adopts an encoder based on a multi-head attention mechanism. After mapping the features of different modalities to a shared dimension, the multi-modal fusion features are generated by calculating attention weights and performing weighted fusion. The global situation map is constructed using a graph neural network, which discretizes the monitoring area into grid nodes. The multimodal fusion features are used as the initial features of the nodes, and the node status is updated through a message passing mechanism based on mean pooling.

[0007] Furthermore, in step S2, the step of obtaining the capability indicators of each robot based on its identity and quantifying them into a structured capability vector includes: The robot's unique device identifier is mapped to the capability database using a hash function. At least one performance indicator is queried and extracted, including drone endurance and hovering accuracy, wheeled robot maximum speed and obstacle crossing ability. After normalizing each indicator, they are concatenated into a fixed-dimensional structured capability vector.

[0008] Furthermore, in step S3, the extraction of high-level risk features includes: using a density-based spatial clustering algorithm to identify the abnormal clustering features of the target, and using the difference method to calculate the instantaneous velocity and acceleration of the target between consecutive frames to identify high-speed maneuver trajectory features; The risk level assessment uses a logistic regression model to classify risk levels.

[0009] Furthermore, in step S3, the matching degree calculation specifically involves: taking the inner product of the robot's capability vector and the task requirement vector to obtain the basic matching degree, and multiplying the basic matching degree by an exponential decay factor based on the distance from the robot's current position to the center of the task area to obtain the scheduling matching score. The target dynamic task allocation adopts a greedy algorithm, iteratively selecting the robot-task pair with the highest scheduling matching score. When the number of available robots is not less than the number of risks to be processed, one-to-one allocation is performed; when the number is insufficient, a multi-robot collaboration mode is activated.

[0010] Furthermore, in step S4, the calculation of the intent score based on the kinematic behavior and posture features of the target includes: calculating the rate of change of velocity through the spatial coordinates of the target in consecutive frames, determining the angle between the target's posture orientation and the protected area, and performing a weighted linear combination of the rate of change of velocity and the cosine value of the angle to obtain a composite intent score. The local risk score is generated by weighted fusion of the composite intent score and the behavioral anomaly degree based on the deviation statistics of movement patterns; the fusion of the local risk score and the global risk assessment value adopts an equal weighted calculation to calculate the joint assessment value.

[0011] Furthermore, in step S4, the initiation of the graded rejection response includes: When the local risk score exceeds the first threshold, an audible and visual warning is triggered. When the second threshold is exceeded, the probability of non-lethal denial measures is corrected by using an exponential decay model in combination with the ambient wind speed, and the execution of denial actions is controlled. When the joint assessment value obtained by fusing the local risk score and the global risk assessment value exceeds the highest rejection threshold, the impact force is calculated based on the mass of the rejection load and the target predicted acceleration, and kinetic rejection is initiated. After execution, the rejection effectiveness is assessed based on the ratio of the actual risk reduction to the expected reduction.

[0012] Furthermore, in step S5, the incremental update of the cross-modal fusion model parameters adopts an online stochastic gradient descent algorithm, uses cross-entropy as the loss function, and iteratively updates the model parameters using the collected feedback data on perception accuracy and rejection effectiveness. The reinforcement learning method uses a time difference approach to update the value table of the task allocation strategy. The experience tuple consists of the current global situation state, the executed scheduling action, the next global situation state, and the immediate reward. The immediate reward is calculated based on a weighted combination of rejection success rate and response latency.

[0013] The present invention also provides a heterogeneous multi-robot cooperative perception and rejection system, comprising: The edge perception and feature extraction module, deployed on each heterogeneous robot node, is used for: S1: Multiple heterogeneous robot nodes synchronously collect environmental data through their onboard multimodal sensors, preprocess and extract features from the collected data locally, generate compressed feature vectors representing the semantics of the environment, and upload them to the central platform. The central integration and situation building module, deployed on the central platform, is used for: S2, the central platform receives feature vectors and raw data uploaded by each heterogeneous robot node, performs cross-modal data association and fusion, generates unified multimodal fusion features and constructs a global situation map covering the monitoring area based on these features; at the same time, based on the identity of each robot, it obtains its capability indicators and quantifies them into structured capability vectors; The risk assessment and task allocation module, deployed on the central platform, is used for: S3, the central platform extracts high-level risk features based on the global situation map, performs risk level assessment and generates task requirements; calculates the matching degree between the capability vector of each robot and the task requirements, and performs attenuation correction based on the spatial distance of the robot's current position relative to the task area to obtain the scheduling matching score, and dynamically allocates tasks with the goal of optimizing the global response time. The graded rejection execution module is deployed on each of the scheduled heterogeneous robot nodes and is used to: S4, the scheduled heterogeneous robot nodes continuously track the designated target, calculate the intent score based on the target's kinematic behavior and posture characteristics, and generate a local risk score by combining the behavior anomaly degree; initiate a hierarchical rejection response based on the local risk score, and initiate the corresponding level of rejection measures when the local risk score and the global risk assessment value issued by the central platform reach the preset conditions. The online learning and strategy optimization module, deployed on the central platform, is used for: S5, the central platform continuously collects feedback data on perception accuracy and rejection effectiveness, updates the parameters of the cross-modal fusion model incrementally, and continuously optimizes the task allocation strategy through reinforcement learning based on the experience tuple consisting of situational state, scheduling actions and effectiveness rewards, forming an iterative closed loop of perception, decision-making, execution and optimization.

[0014] The beneficial effects of this invention are as follows: Through an architecture design combining edge-end compressed sensing and central-end deep fusion, this invention significantly reduces data transmission bandwidth pressure and central computing load while constructing a high-precision global situation map; through structured quantitative expression of heterogeneous robot capabilities and a dynamic matching scheduling mechanism, it achieves efficient collaborative utilization of multiple types of robot resources and optimization of global response time; through composite quantitative evaluation of target intent and a hierarchical rejection strategy combining local and global approaches, it precisely controls rejection intensity, reduces the risk of misjudgment, and ensures progressive and adaptive precise responses to different risk levels; through a closed-loop iterative mechanism driven by incremental learning and reinforcement learning, it endows the system with continuous self-optimization capabilities, enabling the perception model and scheduling strategy to continuously improve with operational accumulation, significantly enhancing the system's long-term adaptability and overall rejection effectiveness in complex monitoring environments. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0017] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0018] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two elements or the interaction between two elements, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0019] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0020] This invention provides a heterogeneous multi-robot collaborative perception and rejection method, comprising the following steps: S1: Multiple heterogeneous robot nodes synchronously collect environmental data through their onboard multimodal sensors, preprocess and extract features from the collected data locally, generate compressed feature vectors representing the semantics of the environment, and upload them to the central platform. In this step, the raw processing of multimodal data is offloaded to the edge, utilizing the local computing power of heterogeneous robot nodes to perform data denoising, spatiotemporal alignment, and semantic feature compression. This results in a significant reduction in the amount of data uploaded to the central platform, effectively alleviating bandwidth pressure on communication links and reducing the computational load and response latency caused by centralized processing of raw data on the central platform. The compressed semantic feature vectors retain the key information needed for environmental understanding, providing a high-quality, low-redundancy data foundation for subsequent cross-modal fusion and situational awareness construction.

[0021] S2, the central platform receives feature vectors and raw data uploaded by each heterogeneous robot node, performs cross-modal data association and fusion, generates unified multimodal fusion features and constructs a global situation map covering the monitoring area based on these features; at the same time, based on the identity of each robot, it obtains its capability indicators and quantifies them into structured capability vectors; In this step, cross-modal association and fusion are used to align and enhance the different modal information collected by heterogeneous robot nodes at the feature level, eliminating the blind spots and uncertainties of single-modal perception and generating semantically consistent and information-complete multimodal fusion features. The global situation map constructed on this basis can reflect the target distribution, environmental dynamics, and abnormal changes within the monitored area in real time, providing comprehensive and accurate situational awareness support for the central platform's macro-level decision-making. Simultaneously, the performance indicators of each heterogeneous robot are quantified into structured capability vectors, enabling a unified and objective expression of the capability differences between different types of robots, providing a calculable and comparable quantitative basis for subsequent task allocation.

[0022] S3, the central platform extracts high-level risk features based on the global situation map, performs risk level assessment and generates task requirements; calculates the matching degree between the capability vector of each robot and the task requirements, and performs attenuation correction based on the spatial distance of the robot's current position relative to the task area to obtain the scheduling matching score, and dynamically allocates tasks with the goal of optimizing the global response time. In this step, high-level semantic features with macro-risk assessment value are abstracted from the global situation map, enabling the system to focus on truly risky events requiring intervention and avoiding unnecessary responses triggered by fluctuations in underlying data. Risk level assessment and task requirement generation transform situational awareness results into clear action instructions, providing clear target inputs for scheduling decisions. The matching degree between capability vectors and task requirements is calculated, and spatial distance attenuation correction is introduced, ensuring that scheduling decisions consider both robot functional fit and spatial accessibility, effectively balancing task execution quality and response time. Ultimately, resource allocation aims at optimizing global response time, maximizing overall collaborative efficiency and avoiding response delays or resource waste.

[0023] S4, the scheduled heterogeneous robot nodes continuously track the designated target, calculate the intent score based on the target's kinematic behavior and posture characteristics, and generate a local risk score by combining the behavior anomaly degree; initiate a hierarchical rejection response based on the local risk score, and initiate the corresponding level of rejection measures when the local risk score and the global risk assessment value issued by the central platform reach the preset conditions. In this step, heterogeneous robot nodes at the execution end are equipped with independent local risk perception and judgment capabilities. They can proactively predict potential risk intentions based on the target's real-time kinematic behavior and posture characteristics, rather than passively waiting for central instructions. The fusion assessment of composite intention scores and behavioral anomaly levels improves the accuracy and timeliness of risk assessment, effectively reducing the probability of false alarms and missed alarms. The tiered rejection response design ensures that the system gradually upgrades response measures according to the risk level, avoiding overuse of rejection methods in low-risk scenarios. The fusion mechanism of local risk scores and the central platform's global risk assessment value balances the real-time advantages of local perception with a holistic view of the global situation, ensuring that rejection decisions are both sensitive and globally coordinated, achieving a precise match between rejection intensity and risk level.

[0024] S5, the central platform continuously collects feedback data on perception accuracy and rejection effectiveness, updates the parameters of the cross-modal fusion model incrementally, and continuously optimizes the task allocation strategy through reinforcement learning based on the experience tuple consisting of situational state, scheduling actions and effectiveness rewards, forming an iterative closed loop of perception, decision-making, execution and optimization.

[0025] This step empowers the system with the ability to automatically learn and evolve from operational experience. Incremental updates to the cross-modal fusion model parameters enable the system to adapt to changes in the monitoring environment and new risk patterns, continuously improving the accuracy of situational awareness. Reinforcement learning optimizes task allocation strategies, allowing the scheduling logic to gradually approach the optimal state through repeated trial and error and evaluation, improving the utilization efficiency and collaborative effectiveness of heterogeneous robot resources. Closed-loop feedback on perception accuracy and rejection effectiveness enables quantitative evaluation and targeted improvement of performance at each stage. This allows the system to move beyond manually preset fixed rules and static models, autonomously adapting to complex and ever-changing deployment scenarios and maintaining long-term stable high performance.

[0026] This invention employs an architecture design that combines edge-end compressed sensing with deep central-end fusion, significantly reducing data transmission bandwidth pressure and central computing load while constructing a high-precision global situation map. Through structured and quantitative representation of heterogeneous robot capabilities and a dynamic matching and scheduling mechanism, it achieves efficient collaborative utilization of multiple robot resources and optimizes global response time. By using composite quantitative evaluation of target intent and a local-global joint hierarchical rejection strategy, it precisely controls rejection intensity, reduces the risk of misjudgment, and ensures progressive and adaptive precise responses to different risk levels. Through a closed-loop iterative mechanism driven by incremental learning and reinforcement learning, it endows the system with continuous self-optimization capabilities, enabling the perception model and scheduling strategy to continuously improve with operational accumulation, significantly enhancing the system's long-term adaptability and overall rejection effectiveness in complex monitoring environments.

[0027] This invention is applied to open or semi-closed scenarios with high security requirements, achieving precise rejection through intelligent collaboration. Utilizing a semi-decentralized architecture combining edge compressed sensing and central deep fusion decision-making, dynamic matching of heterogeneous robot capability vectors, precise quantitative evaluation of target intent, reinforcement learning scheduling, and graded rejection technologies, in large industrial park security scenarios, heterogeneous robot nodes collect real-time environmental data from visible light, infrared, and lidar multimodal sensors. After edge-side denoising, microsecond-level time synchronization, and lightweight neural network dimensionality reduction, the compressed environmental semantic feature vectors are uploaded to the central platform. The central platform generates structured capability vectors for heterogeneous nodes such as drones, wheeled robots, and guard robots through device ID hash mapping. Combined with the abnormal trajectory of intruding targets extracted from the global situation map, the platform calculates the target's velocity change rate and attitude towards the core area to obtain an intent score, which is then determined through inner product matching and distance... The invention employs a distance decay mechanism to schedule optimal robot combinations: UAVs are dispatched for high-altitude tracking and situational information transmission; wheeled robots rapidly approach the target and activate audible and visual warnings; if the target continues to escalate, the central platform adjusts the hit probability based on environmental factors, and dispatches on-duty robots to launch non-lethal denial measures for precise interception. Simultaneously, data on perception accuracy and denial effectiveness are collected throughout the process, and the model and strategy are optimized in a closed loop through incremental learning and reinforcement learning. In border control scenarios, it can achieve all-weather, all-time multi-robot collaborative perception and non-lethal interception against illegal border crossers or suspicious vehicles, effectively avoiding excessive use of force. At the perimeter of large-scale sporting events, exhibitions, and other events, it can quickly respond to sudden crowd surges, efficiently managing the scene through a tiered denial strategy while ensuring the personal safety of participants. This invention has wide applications in smart security, border control, park security, and large-scale event site protection.

[0028] In one embodiment, step S1, the synchronous acquisition of environmental data includes: providing a local time reference through a hardware clock, correcting deviations using a network time protocol, and achieving microsecond-level timestamp alignment between data streams from various modal sensors; The preprocessing includes: applying Gaussian filtering to the visible light image, applying median filtering to the infrared data, removing statistical outliers from the lidar point cloud based on Mahalanobis distance, and normalizing the data for each modality. The feature extraction is achieved through a lightweight convolutional neural network deployed locally on the robot.

[0029] In this embodiment, microsecond-level time synchronization ensures precise temporal matching of multimodal data, providing a reliable alignment benchmark for subsequent cross-modal fusion and fundamentally avoiding perceptual misalignment caused by time deviation. Differentiated filtering and denoising strategies are adopted for the characteristics of each modality data, which can maximize the suppression of interference from different noise sources and improve the quality of the original data. Normalization processing eliminates data scale differences and ensures the consistency of input for feature extraction. Deploying a lightweight convolutional neural network at the edge for semantic feature compression reduces the amount of transmitted data from the original level to the feature level, significantly reducing the bandwidth and computational load of the central platform, while effectively preserving the core spatial information required for environmental understanding. This allows heterogeneous robot nodes to efficiently complete front-end perception tasks even with limited computing power.

[0030] In one embodiment, in step S2, the cross-modal data association and fusion adopts an encoder based on a multi-head attention mechanism. After mapping the features of different modalities to a shared dimension, the multi-modal fusion features are generated by calculating attention weights and performing weighted fusion. The global situation map is constructed using a graph neural network, which discretizes the monitoring area into grid nodes. The multimodal fusion features are used as the initial features of the nodes, and the node status is updated through a message passing mechanism based on mean pooling.

[0031] In this embodiment, cross-modal fusion based on multi-head attention mechanism can dynamically learn the correlation strength between features of different modalities and automatically allocate the optimal fusion weights, so that the multi-modal fusion features can effectively complement the information advantages of each modality and eliminate the perception limitations and uncertainties of a single modality; the graph neural network constructs a global situation map with grid nodes, and realizes the effective aggregation and diffusion of information between nodes through mean pooling message passing, so that the situation map can accurately reflect the spatial distribution and dynamic change patterns of targets within the monitoring area, providing a high-precision and globally consistent situational awareness foundation for high-level risk feature extraction and collaborative task scheduling.

[0032] In one embodiment, step S2, which involves obtaining the capability indicators of each robot based on its identity and quantifying them into a structured capability vector, includes: The robot's unique device identifier is mapped to the capability database using a hash function. At least one performance indicator is queried and extracted, including drone endurance and hovering accuracy, wheeled robot maximum speed and obstacle crossing ability. After normalizing each indicator, they are concatenated into a fixed-dimensional structured capability vector.

[0033] In this embodiment, a hash function is used to achieve a fast mapping from device identification codes to the capability database, ensuring a reliable association and efficient retrieval of heterogeneous robot identities and their capability parameters. The differentiated performance indicators of various types of robots are normalized, eliminating the incomparability caused by different units of measurement. This allows the capabilities of heterogeneous nodes such as drones and wheeled robots to be uniformly quantified into standardized structured vectors, providing a fair and computable matching basis for subsequent task allocation and effectively supporting the accurate quantitative evaluation of heterogeneous resources in scheduling decisions.

[0034] In one embodiment, step S3, the extraction of high-level risk features includes: using a density-based spatial clustering algorithm to identify the abnormal clustering features of the target, and using the difference method to calculate the instantaneous velocity and acceleration of the target between consecutive frames to identify high-speed maneuver trajectory features; The risk level assessment uses a logistic regression model to classify risk levels.

[0035] In this embodiment, the density-based spatial clustering algorithm can automatically detect spatial distribution anomalies of targets within the monitoring area without relying on preset risk templates, adapting to changing risk scenarios and effectively identifying potential risk patterns such as abnormal clustering. The differential method captures instantaneous changes in target motion through precise calculation of motion parameters between consecutive frames, realizing a quantitative description of high-speed maneuver trajectories and improving sensitivity to dynamic risks. The combination of the two features constructs a complete risk feature expression from both spatial and motion dimensions, making risk assessment more comprehensive and reliable. The logistic regression model maps multidimensional risk features to probabilistic level classifications, outputting continuous and quantifiable risk levels, which facilitates the generation of task requirements that match the risk level, improving task allocation granularity and response accuracy.

[0036] In one embodiment, in step S3, the matching degree calculation specifically involves: taking the inner product of the robot's capability vector and the task requirement vector to obtain the basic matching degree, and multiplying the basic matching degree by an exponential decay factor based on the distance from the robot's current position to the center of the task area to obtain the scheduling matching score. The target dynamic task allocation adopts a greedy algorithm, iteratively selecting the robot-task pair with the highest scheduling matching score. When the number of available robots is not less than the number of risks to be processed, one-to-one allocation is performed; when the number is insufficient, a multi-robot collaboration mode is activated.

[0037] In this embodiment, the robot's capabilities and task requirements are finely matched in a multi-dimensional feature space through inner product operations, quantifying the degree of fit between the two and ensuring that scheduling decisions are based on objectively computable adaptability. A distance decay factor is introduced to correct the matching degree, taking into account both the response time and spatial reachability constraints of task execution, avoiding response delays caused by long-distance scheduling, and making the scheduling matching score more accurately reflect the actual reachability. The application of a greedy algorithm ensures that the global response time is close to optimal while keeping the time complexity of the combinatorial optimization problem within a real-time running range, adapting to large-scale dynamic task allocation scenarios. Based on the comparison of available resources and the number of risks, the system adaptively switches between one-to-one allocation and multi-robot collaboration modes. This ensures that each task obtains a dedicated execution unit when resources are abundant, and that all risks are effectively covered through the collaboration mode when resources are scarce, significantly improving the resource utilization efficiency and global response capability of the heterogeneous multi-robot system in complex multi-task scenarios.

[0038] In one embodiment, step S4, the calculation of the intent score based on the kinematic behavior and posture features of the target, includes: calculating the rate of change of velocity through the spatial coordinates of the target in consecutive frames, determining the angle between the target's posture orientation and the protected area, and performing a weighted linear combination of the rate of change of velocity and the cosine value of the angle to obtain a composite intent score. The local risk score is generated by weighted fusion of the composite intent score and the behavioral anomaly degree based on the deviation statistics of movement patterns; the fusion of the local risk score and the global risk assessment value adopts an equal weighted calculation to calculate the joint assessment value.

[0039] In this embodiment, a weighted combination of the rate of change of velocity and the cosine of the angle between the target and the protected area is used to characterize the target's potential intention from two dimensions: the intensity of the movement and its spatial directionality. Compared with a single kinematic index, this approach can more accurately distinguish between normal movement and risky behavior with a clear direction, effectively improving the accuracy and lead time of risk intention prediction. The composite intention score is fused with the behavior anomaly degree based on the statistical deviation of the movement pattern to generate a local risk score. This allows the risk assessment to take into account both the target's current intention and long-term behavioral pattern deviations, reducing misjudgments triggered by occasional actions. The local risk score and the global risk assessment value are fused with equal weights. This ensures the timeliness of the autonomous response of heterogeneous robot nodes while avoiding decision-making biases caused by limited local vision. It achieves an effective balance between local agility and global consistency, ensuring that the joint assessment value can more comprehensively and robustly reflect the target's true risk level, providing a reliable decision-making basis for the accurate triggering of graded rejection.

[0040] In one embodiment, step S4, the initiation of the graded rejection response includes: When the local risk score exceeds the first threshold, an audible and visual warning is triggered. When the second threshold is exceeded, the probability of non-lethal denial measures is corrected by using an exponential decay model in combination with the ambient wind speed, and the execution of denial actions is controlled. When the joint assessment value obtained by fusing the local risk score and the global risk assessment value exceeds the highest rejection threshold, the impact force is calculated based on the mass of the rejection load and the target predicted acceleration, and kinetic rejection is initiated. After execution, the rejection effectiveness is assessed based on the ratio of the actual risk reduction to the expected reduction.

[0041] In this embodiment, a hierarchical denial system with precise matching between risk and response intensity is constructed by setting progressive multi-level thresholds. In the low-risk stage, non-contact audible and visual warnings are prioritized for deterrence to minimize substantial damage to the target, reflecting the core concept of non-lethal denial. When the risk escalates, environmental wind speed is introduced to correct the hit probability of non-lethal denial measures, ensuring that denial execution takes into account the dynamic impact of environmental factors and guarantees the effectiveness and accuracy of non-lethal measures under complex weather conditions. When the risk reaches the highest level, the impact force is accurately calculated based on the denial payload mass and the target's predicted acceleration to ensure that the physical impact force of kinetic denial is commensurate with the level of risk, avoiding excessive use of force. After each denial execution, the ratio of the actual risk reduction to the expected reduction is evaluated in real time, providing closed-loop feedback for whether to escalate or adjust the denial strategy. This enables denial actions to have dynamic, assessable, traceable, and optimizable continuous improvement capabilities, fully supporting a precise and progressive non-lethal denial decision-making system.

[0042] In one embodiment, in step S5, the cross-modal fusion model parameters are updated incrementally using an online stochastic gradient descent algorithm with cross-entropy as the loss function, and the collected perception accuracy and rejection effectiveness feedback data are used to iteratively update the model parameters. The reinforcement learning method uses a time difference approach to update the value table of the task allocation strategy. The experience tuple consists of the current global situation state, the executed scheduling action, the next global situation state, and the immediate reward. The immediate reward is calculated based on a weighted combination of rejection success rate and response latency.

[0043] In this embodiment, an online stochastic gradient descent algorithm is used for incremental parameter updates, eliminating the need to retrain the entire historical dataset. This significantly reduces the computational overhead and storage costs of model optimization, enabling the system to smoothly adapt to environmental changes and new risk patterns during continuous operation. The cross-entropy loss function exhibits excellent gradient characteristics at the classification boundary, effectively guiding the convergence direction of model parameters and accelerating the correction process of perception errors by the fusion model. By directly utilizing real-time feedback data on perception accuracy and rejection effectiveness collected during operation to drive model iteration, model optimization is highly coupled with actual system performance indicators. This ensures that improvements in the quality of fusion features can be directly translated into improvements in situational awareness accuracy and rejection decision effectiveness, forming a data-driven self-evolutionary closed loop.

[0044] The present invention also provides a specific embodiment, including: S1: Multiple heterogeneous robot nodes synchronously collect environmental data through their onboard multimodal sensors. The collected data is preprocessed and features are extracted locally to generate compressed feature vectors that represent the semantics of the environment, which are then uploaded to the central platform.

[0045] The heterogeneous robot node activates a multimodal sensor array, including a visible light camera, an infrared thermal imager, and a lidar, to synchronously acquire environmental data. Microsecond-level timestamp alignment between the data streams from each sensor is achieved using a hardware clock and network time protocol, and the raw data undergoes filtering, noise reduction, and standardization preprocessing. Subsequently, the processed data is input into a lightweight convolutional neural network deployed locally on the robot. This network extracts basic features through convolutional layers, performs nonlinear mapping via activation functions, and then compresses the features using pooling layers, ultimately outputting a fixed-dimensional environmental semantic feature vector. This step completes the initial edge processing and dimensionality reduction of the data, and the feature vector is uploaded to the central platform in real time, effectively reducing the network transmission bandwidth pressure.

[0046] Specifically, the heterogeneous robot nodes activate a multimodal sensor array, including a visible light camera, an infrared thermal imager, and a lidar, to synchronously acquire environmental data. Microsecond-level timestamp alignment between the data streams from each sensor is achieved using a hardware clock and network time protocol, and the raw data undergoes filtering, noise reduction, and standardization preprocessing. Subsequently, the processed data is input into a lightweight convolutional neural network deployed locally on the robot. This network extracts basic features through convolutional layers, performs nonlinear mapping via activation functions, and then compresses the features using pooling layers, ultimately outputting a fixed-dimensional environmental semantic feature vector. This step completes the initial edge processing and dimensionality reduction of the data, and the feature vector is uploaded to the central platform in real time, effectively reducing the network transmission bandwidth pressure.

[0047] In one embodiment, the synchronous acquisition of environmental data includes: providing a local time reference through a hardware clock, correcting deviations using a network time protocol, and achieving microsecond-level timestamp alignment between data streams from various modal sensors; the preprocessing includes: applying Gaussian filtering to visible light images, applying median filtering to infrared data, removing statistical outliers from lidar point clouds based on Mahalanobis distance, and normalizing the data from each modality; the feature extraction is achieved through a lightweight convolutional neural network deployed locally on the robot, which sequentially includes a 3×3 convolutional layer, a ReLU activation function, and a 2×2 max-pooling layer, outputting a fixed-dimensional environmental semantic feature vector, and the network parameter size does not exceed 1M.

[0048] The specific implementation method of this step will be explained in detail below.

[0049] The heterogeneous robot node initializes the multimodal sensor array, ensuring the simultaneous startup of the visible light camera, infrared thermal imager, and lidar. The visible light camera captures RGB images at 1920x1080 resolution and 30fps; the infrared thermal imager operates in the 8-14μm band, outputting thermal distribution maps; the lidar uses a 905nm wavelength to generate point cloud data with an accuracy of ±2cm. The sensors are synchronized with the Network Time Protocol (NTP) via a hardware clock (such as a PTP hardware clock). The hardware clock provides a local time reference, and the NTP protocol corrects for network latency, ensuring timestamp alignment accuracy at the microsecond level. The timestamp alignment formula is: ; in, It is the sensor's original timestamp. This is the NTP correction bias, calculated using the least squares method: ; It serves as a reference time source. This mechanism eliminates timing drift between sensors and improves the reliability of multimodal data fusion.

[0050] The raw sensor data is filtered and denoised. Gaussian filtering is applied to visible light images to remove noise; the formula is: ; in This is the standard deviation (set to 1.5). x and y Pixel coordinates.

[0051] Infrared thermal imagers often output pulse interference such as salt-and-pepper noise in their thermal distribution maps. Median filtering can effectively remove this type of noise while preserving image edge information relatively well. Therefore, for infrared data median filtering, with a window size of 3×3, the formula is: ; in Given an input image matrix, To output the filtered image in coordinates Pixel value at that location, and This represents the row and column offsets within the window relative to the center pixel.

[0052] LiDAR point clouds are used for statistical outlier removal, with the threshold based on Mahalanobis distance. ; p It is a vector of point coordinates, and μ is the mean of the neighborhood. σ It is a covariance matrix, and points with a distance greater than 3 times the standard deviation are removed.

[0053] After the aforementioned differential filtering and denoising process, the numerical ranges and dimensions of the three modal data—visible light images, infrared thermal images, and lidar point clouds—differ significantly. Directly inputting these into subsequent networks would affect the stability and convergence speed of feature extraction. To eliminate scale differences between and within modalities and ensure the consistency of input data, all modal data undergo standardization preprocessing, normalizing the data to the [0,1] range. The formula... , x It is the original value. It is the minimum value of the data. It is the maximum value of the data. x' The normalized values. Through this process, the modal data are uniformly mapped to the same dimension range, ensuring input consistency and accelerating subsequent processing.

[0054] The processed data is input into a lightweight convolutional neural network (CNN) deployed locally on the robot. The network architecture is designed so that the input layer receives multimodal data (images and point clouds converted into 2D feature maps), and the convolutional layers use 3x3 kernels with a stride of 1 and the same padding. The formula is: ; in, It is the weight matrix of the l-th convolutional kernel. It is a bias term. It is the feature map of the previous layer. This indicates a convolution operation.

[0055] Convolution is essentially a linear transformation. If a network consists only of stacked convolutional layers, regardless of its depth, its expressive power is still limited to linear mapping and cannot fit the complex nonlinear patterns in environment-aware tasks. To address this, the ReLU activation function is introduced to perform a nonlinear mapping on the convolutional output. This approach maintains computational simplicity while endowing the network with nonlinear expressive power and effectively alleviates the gradient vanishing problem.

[0056] The activation function used is ReLU, and the formula is as follows: This function sets negative inputs to zero while keeping positive inputs unchanged, thus achieving non-linear mapping and suppressing negative values.

[0057] After feature extraction via convolutional layers and ReLU nonlinear activation, the feature map still retains a high dimensionality, and directly inputting it into subsequent layers would impose a significant computational burden. Max pooling layers downsample the local neighborhood, compressing the spatial size of the feature map while preserving salient feature responses, reducing the number of parameters and computational cost, and enhancing the robustness of features to small displacements. Max pooling is applied in the pooling layer, with a window size of [missing value]. The step size is 2, and the formula is: ; in Pooling output feature map at position The value at that location, and Representing the output position respectively The corresponding input feature map row index set and column index set, For the input feature map at location The eigenvalue at that location. R and C It is a pooling region that compresses the feature dimension.

[0058] The network outputs a fixed 512-dimensional environmental semantic feature vector, such as This lightweight design requires only 1M parameters, reducing edge computing load while preserving critical spatial information.

[0059] Feature vectors are uploaded to the central platform in real time via the robot's local network interface, using UDP with QoS to ensure low latency. The original data volume is reduced from GB to KB (e.g., from 100MB / s to 10KB / s), significantly alleviating network bandwidth pressure. The beneficial effects of this step include a timestamp alignment mechanism combined with a hardware clock and NTP, improving multimodal data synchronization accuracy to the microsecond level and avoiding fusion errors; and a lightweight CNN compressing features through pooling layers, efficiently reducing dimensionality at the edge, decreasing the computational burden on the central platform, and supporting real-time response.

[0060] S2, the central platform receives feature vectors and raw data uploaded by each heterogeneous robot node, performs cross-modal data association and fusion, generates unified multimodal fusion features, and constructs a global situation map covering the monitoring area based on these features; at the same time, based on the identity of each robot, it obtains its capability indicators and quantifies them into structured capability vectors.

[0061] The central platform receives feature vectors and raw sensor data streams from all heterogeneous robot nodes. First, it runs cross-modal data association and fusion algorithms, such as an attention-based encoder, to align and fuse features from different modalities at the underlying level, generating a unified multimodal fusion feature. Based on this, a global situational awareness map covering the entire monitoring area is constructed. Simultaneously, the system uses a hash function to quickly map each robot's unique device identification code to a capability database, querying and extracting its type-related performance indicators, such as the endurance and hovering accuracy of drones, and the maximum speed and obstacle-crossing capability of wheeled robots. After normalizing these indicators, they are concatenated into a standardized structured capability vector, providing a quantitative basis for subsequent collaborative scheduling.

[0062] Specifically, the central platform receives data streams from all heterogeneous robot nodes via the TCP / IP protocol, including lightweight feature vectors and raw sensor data (such as visible light images, infrared thermal maps, and LiDAR point clouds). The data streams are accompanied by timestamps and unique device identifiers (IDs) for each robot to ensure time alignment.

[0063] The following provides a detailed explanation of the specific implementation methods for each sub-step.

[0064] (I) Cross-modal data association and fusion Because visible light, infrared, and lidar sensors differ fundamentally in their physical imaging mechanisms and data representation formats, simply stitching them together or processing them independently will lead to fragmented information between the modes, making it difficult to form a unified understanding of the environment. Therefore, the system runs a cross-modal data association and fusion algorithm, employing an encoder architecture based on a multi-head attention mechanism. By dynamically learning the association strength between modes, it achieves feature alignment and complementary fusion at the underlying level. The feature vectors of each mode (e.g., visual features)... Infrared features LiDAR characteristics First, it is mapped to a shared dimension through a linear projection layer. ,get , , ,in This is a trainable weight matrix.

[0065] Mapping features from different modalities to a shared dimensional space eliminates dimensional differences between modalities, enabling features from different sources to be compared and fused in a unified vector space. Based on this, cross-modal attention weights are calculated for the query vector. (Initialized to modal average), key vector Sum value vector From various modes Attention score ,in This is a scaling factor used to prevent the gradient of the softmax function from vanishing due to excessively large inner product values. The attention weights are obtained after normalization: Attention weight This reflects the relative importance of each modal feature to the overall perception task in the current environment, and the system fuses features accordingly. This mechanism achieves underlying alignment through dynamic weight allocation, effectively handles modal differences, improves fusion accuracy by approximately 15%, and helps reduce false associations.

[0066] (II) Construction of the Global Situation Map Generate unified multimodal fusion features Subsequently, to construct a global situational awareness covering the entire monitoring area, rather than being limited to local observations from individual heterogeneous robot nodes, the system inputs a situational map construction module. This module uses a graph neural network (GNN) to discretize the monitoring area into grid nodes. Each node is associated with spatial coordinates. Feature fusion. As initial features of nodes, graph neural networks enable grid nodes to aggregate neighborhood information through information exchange between nodes, thereby forming a continuous and complete situational representation. Specifically, node states are updated through message passing: ; in It is the ReLU activation function. These are neighboring nodes; AGGREGATE uses mean pooling. and They are nodes i In the t Second and third t The state vector after +1 iterations. W and U Learnable weight matrix. Outputs a global situation map. It covers the entire area and reflects the distribution of targets and environmental changes in real time.

[0067] (III) Construction of Heterogeneous Robot Capability Vectors In multi-robot collaborative systems, different types of robots have different strengths in terms of mobility, payload capacity, and perception accuracy. Without a unified, quantifiable representation of these capabilities, task allocation will struggle to accurately match the actual execution capabilities of each robot. Therefore, the system processes device identification codes. Each robot ID (e.g., the string "Drone_001") is determined using SHA... A 256-bit hash function, h=SHA256ID, maps ID to a 32-bit hash value for indexing and querying in an in-memory database (such as Redis). Using a hash function for mapping enables identity retrieval in constant time complexity, ensuring the responsiveness of real-time scheduling for large-scale robot clusters. Database storage devices (such as drones and wheeled robots) and performance metrics are used to extract specific parameters: drone metrics include flight time (bbat) (minutes) and hovering accuracy (phover) (meters); wheeled robot metrics include maximum speed (smax) (m / s) and obstacle crossing capability (oobs) (centimeters). Since these metrics have different physical dimensions and value ranges, directly using them in subsequent matching calculations would lead to dimensional differences dominating the matching results, rendering them incomparable. Therefore, these metrics are minified. Max normalization: ; in This is the global minimum value calculated from historical data. This is the global maximum value calculated from historical data. These are the original indicator values. This is the normalized index value. Normalization eliminates dimensional differences and ensures that the index is in the [0,1] interval, facilitating fair quantification.

[0068] Finally, the normalized indices are concatenated into a structured capability vector. For example, drone vectors: ; Wheeled robot vector: ; Vector Dimension The value is fixed at 4 (based on the maximum number of indicators), and missing values ​​are filled with 0. Through the above processing, the capabilities of different types of heterogeneous robots are uniformly expressed as fixed-dimensional, dimensionless structured vectors. The output standardized vectors are stored in shared memory, providing a quantitative basis for S3 collaborative scheduling and improving scheduling efficiency by 20%.

[0069] S3, the central platform extracts high-level risk features based on the global situation map, performs risk level assessment and generates task requirements; calculates the matching degree between the capability vector of each robot and the task requirements, and performs attenuation correction based on the spatial distance of the robot's current position relative to the task area to obtain a scheduling matching score, and dynamically allocates tasks with the goal of optimizing the global response time.

[0070] Based on the established global situation map, the central platform runs an advanced feature extraction module, focusing on identifying and extracting high-level local key situation features, such as abnormal clusters and high-speed maneuver trajectories. These features specifically serve macro-risk assessment. The system performs an initial risk level judgment based on the extracted risk features and generates corresponding task requirement vectors. Subsequently, the collaborative task allocation engine is activated: this engine performs an inner product operation on the capability vectors of each robot and the task requirement vectors to obtain a basic matching degree; this is then multiplied by an exponential decay factor based on the distance from the robot's current position to the task point to calculate the final scheduling matching score. With the goal of optimizing global response time, the system dynamically decides and issues scheduling instructions, comprehensively considering the total number of available robots and the number of risk targets to be processed, assigning tasks such as monitoring, interception, or attack to the most suitable heterogeneous robots.

[0071] Specifically, the central platform loads a global situation map generated by S2, which includes spatiotemporal coordinates, target trajectory, and environmental semantic information. Based on this, the system first extracts high-level risk features from the global situation to serve macro-risk assessment, then performs a quantitative assessment of the risk level and generates a task requirement vector accordingly, and finally completes the optimal matching and scheduling of heterogeneous robot nodes and tasks through a collaborative task allocation engine.

[0072] In one embodiment, the extraction of high-level risk features includes: using a density-based spatial clustering algorithm to identify abnormal clustering features of targets, and using a difference method to calculate the instantaneous velocity and acceleration of targets between consecutive frames to identify high-speed maneuver trajectory features; the risk level assessment is combined with a logistic regression model to classify the risk level.

[0073] In another embodiment, the matching degree calculation specifically involves: taking the inner product of the robot's capability vector and the task requirement vector to obtain the basic matching degree; multiplying the basic matching degree by an exponential decay factor based on the distance from the robot's current position to the center of the task area to obtain the scheduling matching score; the target dynamic task allocation adopts a greedy algorithm, iteratively selecting the robot-task pair with the highest scheduling matching score; performing one-to-one allocation when the number of available robots is not less than the number of risks to be processed; and starting a multi-robot collaboration mode when the number of available robots is insufficient.

[0074] The following provides a detailed explanation of the specific implementation methods for each sub-step.

[0075] (a) Extraction of high-level risk characteristics: While the global situation map contains rich environmental information, much of its detail belongs to low-level data features and does not directly serve risk assessment. Making scheduling decisions directly based on raw situation data is not only computationally inefficient but also prone to triggering unnecessary responses due to data noise. Therefore, the system runs an advanced feature extraction module to abstract high-level semantic features with macro-level risk assessment value from the global situation map. Density-based spatial clustering algorithms (such as DBSCAN) are used to identify anomalous clustering features. By calculating the target density deviation value within a local area, if it exceeds a threshold... (Unit: number of targets / square meter) If any clustering is detected, it is marked as an abnormal cluster. Simultaneously, for high-speed maneuver trajectories, the module extracts the target position sequence of continuous time frames and applies the finite difference method to calculate the instantaneous velocity. and acceleration The formula is and ,in Indicates the target in time coordinate vector, The sampling interval is specified (default 0.1 seconds). Velocity reflects the target's current motion state, while acceleration characterizes the instantaneous change in motion state; the latter is particularly crucial for identifying risky behaviors such as sudden acceleration and sprinting. If the trajectory persists for more than 3 frames, it is identified as a high-speed maneuver trajectory. The two types of features described above characterize risk patterns from two dimensions: spatial distribution anomalies and motion pattern anomalies. They are specifically designed for macro-level risk assessment, distinct from the low-level data alignment features in S2 that serve situation building, ensuring the independence of high-level situation analysis. The extraction process runs on GPU acceleration, with a processing time controlled within 50ms, and the output is a structured feature vector. ; in To score the aggregation density, The rating is based on the maneuverability.

[0076] (II) Risk Level Assessment and Task Requirements Generation High-level risk feature vector The system quantifies the anomaly of targets from two dimensions: spatial aggregation and movement maneuverability. However, relying solely on a threshold from a single dimension is insufficient for a comprehensive assessment of risk levels. Therefore, the system is based on... The initial risk level assessment uses a three-tiered classification logic: if or If the normalized value is true, the risk level is high; otherwise, the probability value output by the logistic regression model is used to classify it into medium and low levels. The logistic regression model maps multidimensional risk features to continuous probability values, outputting smooth and quantifiable risk level assessment results, which facilitates the fine-grained generation of subsequent task requirement vectors. The task requirement vector rj is dynamically generated based on the risk level; for example, a high-risk task corresponds to... The components represent the weight requirements for monitoring, interception, and attack, respectively. The weight values ​​are obtained based on historical data training to ensure that the vector dimension is consistent with the robot capability vector generated in S2. The generation logic is embedded in a real-time decision-making loop with a latency of less than 10ms, supporting the processing of 100+ task requests per second.

[0077] (III) Collaborative Task Allocation and Scheduling After generating the task requirement vector, each task needs to be matched to the most suitable heterogeneous robot node for execution. Assigning tasks solely based on robot capabilities or spatial distance will lead to capability mismatches or excessive response delays. Therefore, after the collaborative task allocation engine starts, it retrieves the pre-generated capability vector from S2. Each robot Vectors such as: ; Standardize to the range [0,1]. The engine calculates the basic matching degree mij through inner product operation: The essence of inner product operation is to measure the directional consistency and magnitude matching degree between the robot's capability vector and the task requirement vector in a multi-dimensional feature space, thus quantifying the fit between the robot's capabilities and task requirements. Subsequently, a distance decay factor is introduced, based on the robot's current position... Arrive at the mission point Euclidean distance Calculate the exponential decay term ,in The attenuation factor was experimentally calibrated to balance the influence of distance. As the distance between the robot and the task area increases, the exponential attenuation factor causes the scheduling matching score to decrease smoothly, ensuring that robots with similar capabilities are prioritized for deployment, balancing task execution quality and response time. Final scheduling matching score. for: This design, which combines inner product and attenuation, avoids deviations in pure capability or distance, thus improving allocation efficiency. The beneficial effect is a significant reduction in response latency, with measured improvements in matching accuracy of 15% in dense scenarios.

[0078] The system aims to minimize global response time. For the target, define: ; in, For the task Completion time, This represents the total number of risk targets to be processed. In multi-task, multi-robot scenarios, exhaustively enumerating all possible allocation schemes will lead to a combinatorial explosion problem, which cannot meet real-time scheduling requirements. Therefore, the dynamic decision-making module comprehensively considers the total number of available robots. R and A greedy algorithm is used to optimize the allocation: initialize the air conditioning degree set, and iteratively select the highest degree. robots Task pair, with the following constraints: Ensure one-to-one allocation; otherwise, multiple robots must collaborate. Optimize the process time complexity. Executed on the edge computing unit, it outputs scheduling instructions such as "Drone performs monitoring, wheeled robot intercepts". The decision results are sent through a low-latency communication protocol, ensuring that the instruction transmission time is <20ms, thus achieving accurate task allocation for heterogeneous robots.

[0079] S4, the scheduled heterogeneous robot nodes continuously track the designated target, calculate the intent score based on the target's kinematic behavior and posture characteristics, and generate a local risk score by combining the behavior anomaly degree; initiate a hierarchical rejection response based on the local risk score, and initiate the corresponding level of rejection measures when the local risk score and the global risk assessment value issued by the central platform reach the preset conditions.

[0080] The robot, dispatched to the task area, begins autonomous execution. It continuously tracks the designated target, calculating its velocity change rate by analyzing the target's spatial coordinates across consecutive frames, and combining this with its orientation relative to the protected area to calculate a quantified composite intent score. This intent score is combined with other behavioral anomalies of the target to generate a local risk score. If the risk score exceeds a first threshold, an audible and visual warning is automatically triggered for deterrence. If the target's behavior continues to deteriorate and the risk score exceeds a higher threshold, the system calculates the hit probability correction for non-lethal denial measures based on environmental factors (such as wind speed) and controls the robot to fire net guns or stun guns for interception. When the risk escalates further, the robot merges its local risk score with a risk assessment value based on the global situation issued by the central platform to obtain a joint risk assessment value. If this value exceeds the highest denial threshold, the required impact force is calculated based on the mass and predicted acceleration of the kinetic energy projectile, initiating precise kinetic energy projectile denial. The denial effectiveness is immediately evaluated after each action, providing feedback for deciding whether to escalate denial measures.

[0081] Specifically, the heterogeneous robot nodes dispatched to the task area begin to execute autonomously, collecting spatial coordinate data of the target in real time through onboard multimodal sensors (such as RGB-D cameras) and continuously tracking the designated target.

[0082] In one embodiment, the calculation of the intent score based on the target's kinematic behavior and posture features includes: calculating the rate of change of velocity using the target's continuous frame spatial coordinates, determining the angle between the target's posture orientation and the protected area, and performing a weighted linear combination of the rate of change of velocity and the cosine value of the angle to obtain a composite intent score; the local risk score is generated by weighted fusion of the composite intent score and the behavior anomaly degree based on motion pattern deviation statistics; the fusion of the local risk score and the global risk assessment value uses an equal-weighted calculation to obtain a joint assessment value.

[0083] In another embodiment, the activation of the graded denial response includes: triggering an audible and visual warning when the local risk score exceeds a first threshold; when it exceeds a second threshold, correcting the hit probability of the non-lethal denial method using an exponential decay model based on the ambient wind speed, and controlling the execution of the denial action; when the joint assessment value obtained by fusing the local risk score and the global risk assessment value exceeds the highest denial threshold, calculating the impact force based on the mass of the denial load and the target predicted acceleration, activating kinetic denial, and evaluating the denial effectiveness after execution based on the ratio of the actual risk reduction to the expected reduction.

[0084] The following provides a detailed explanation of the specific implementation methods for each sub-step.

[0085] (a) Target tracking and intent assessment The robot uses onboard multimodal sensors (such as RGB) A D-camera acquires spatial coordinate data of the target in real time across consecutive frames, and Kalman filtering is used for noise suppression and data smoothing. Target position sequence timestamp t The coordinates are obtained from the following data, where the coordinate unit is meters and the time interval is... Set the time to 0.1 seconds. Calculate the instantaneous velocity vector. Rate of change of velocity (magnitude of acceleration) ,here It represents the Euclidean norm, which quantifies the intensity of the target's motion.

[0086] Relying solely on the rate of change of velocity is insufficient to distinguish whether a target's motion is a purposeful approach towards the protected area or a risk-free parallel passage. Therefore, target attitude and orientation information is incorporated for joint judgment. Attitude and orientation angle Calculated by a target attitude estimation model (such as vector analysis based on skeleton key points), it is defined as the angle between the target's forward direction vector and the center vector of the protected area, ranging from 0 to 180 degrees. A smaller value indicates that the target is directly facing the protected area, and the higher the risk intent. By weighting the rate of change of velocity and the attitude orientation angle, a composite intent score is obtained, comprehensively characterizing the target's potential intent from two dimensions: the intensity of motion and spatial directionality. The following is obtained through a weighted linear combination: ; Weight and Based on experience, the values ​​are set to 0.6 and 0.4 (which satisfy the requirements). ), The function ensures a positive increase in score when the target moves toward the protected area. This method dynamically quantifies potential risk intent, improves early warning accuracy, and avoids misjudgments. Behavioral anomaly score B is extracted from other sensors (such as millimeter-wave radar) and calculated based on the statistical deviation of the target's motion pattern from its normal trajectory, for example, using Mahalanobis distance. The composite intent score reflects the target's current instantaneous risk tendency, while the behavioral anomaly score characterizes the degree to which its long-term behavioral pattern deviates from the normal state. The fusion of these two scores balances immediacy and statistical accuracy, generating a more robust risk assessment. Local risk score. The generating formula is ,coefficient and Take values ​​of 0.7 and 0.3 respectively, and normalize to 0. Range of 100.

[0087] (ii) Tiered denial response To ensure that appropriate rejection measures are adopted for different risk levels and to avoid over- or under-response caused by a single threshold trigger, the system has established a progressive, tiered rejection mechanism. ( =30 (the first threshold) triggers the audible and visual warning module, outputting a high-brightness strobe light and an alarm sound. This stage employs non-contact deterrence, intervening in the target's behavior with minimal force, embodying the core concept of non-lethal denial.

[0088] If the risk score continues to deteriorate and ( =60 (a higher threshold) indicates that the audible and visual warnings have failed to effectively deter the target, and the risk level has risen to the point where physical intervention is required; however, considering that environmental factors can significantly affect the effectiveness of non-lethal denial measures, failure to make corrections will lead to hit deviations. Therefore, the system incorporates environmental wind speed. (Unit: m / s) Corrected base hit probability of non-lethal denial measures . Correction formula: ; in, =0.1 is the wind resistance attenuation coefficient. The initial value is 0.9; this exponential decay model effectively compensates for environmental interference and improves interception accuracy. The controller fires either a net gun or an electric shock device, with the firing angle calculated based on the predicted target position.

[0089] (III) Joint Risk Assessment and Kinetic Denial When the risk escalates, the robot receives a global risk assessment value from the central platform. (range 0) 100). To balance the real-time sensitivity of local perception in heterogeneous robot nodes with the overall global situational awareness of the central platform, and to avoid decision-making biases caused by unilateral reliance, the system integrates local risk scores and global risk assessment values ​​with equal weights to generate a joint risk assessment value: ; Fusion weights = 0.5 and = 0.5 ensures global and local balance. If ( =80 is the highest denial threshold), indicating that the target poses a highly certain and serious risk, requiring the initiation of precise physical denial measures. Based on the mass of the kinetic energy projectile. m (unit: kg) and predicted acceleration (Derived from the target motion model) Calculate the required impact force By precisely calculating the impact force, a symmetrical match is achieved between the rejection strength and the level of risk, allowing for precise control of the launcher's output force. Effectiveness is assessed immediately after rejection, for example, by calculating effectiveness metrics using impact detection sensors. ;like E <0.7, feedback decision-making system upgrade rejection measures. Form a closed-loop adjustment chain of execution-evaluation-feedback.

[0090] S5, the central platform continuously collects feedback data on perception accuracy and rejection effectiveness, updates the parameters of the cross-modal fusion model incrementally, and continuously optimizes the task allocation strategy through reinforcement learning based on the experience tuple consisting of situational state, scheduling actions and effectiveness rewards, forming an iterative closed loop of perception, decision-making, execution and optimization.

[0091] The central platform, acting as the optimization hub, continuously collects perception data accuracy reports and effectiveness feedback data from the entire network for each rejection action. On one hand, this data is used to incrementally learn and update the parameters of the multimodal fusion model in S2 online, improving the accuracy of environmental understanding. On the other hand, each task allocation (scheduling action) and its corresponding rejection effectiveness (reward) under the corresponding environmental situation (state) are constructed into an experience tuple and input into a purely logic-driven reinforcement learning framework. This framework uses a temporal difference method to continuously update its task allocation strategy's value evaluation table, thereby optimizing the scheduling decision logic. The optimized model parameters and strategy table are periodically or triggered and distributed to each heterogeneous robot node, completing a full closed loop from perception, decision-making, execution to evaluation and optimization, driving iterative improvement in the overall system performance.

[0092] Specifically, the central platform acts as the optimization hub, receiving feedback data from various heterogeneous robot nodes in real time through a dedicated communication protocol, and continuously iterating and optimizing the system model and strategies.

[0093] In one embodiment, the incremental update of the cross-modal fusion model parameters employs an online stochastic gradient descent algorithm, using cross-entropy as the loss function, and iteratively updates the model parameters using the collected feedback data on perception accuracy and rejection effectiveness. The reinforcement learning method uses a time difference approach to update the value table of the task allocation strategy. The experience tuple consists of the current global situation state, the executed scheduling action, the next global situation state, and the immediate reward. The immediate reward is calculated based on a weighted combination of rejection success rate and response latency.

[0094] The following provides a detailed explanation of the specific implementation methods for each sub-step.

[0095] (a) Feedback Data Collection The central platform receives real-time perception data accuracy reports from heterogeneous robot nodes via a dedicated communication protocol, including target detection accuracy. and false alarm rate And denial performance feedback data, such as block success rate. and response delay These data are stored in a structured format in a central database, ensuring low latency and high integrity. Continuous, structured data collection lays the foundation for subsequent online optimization of models and strategies.

[0096] (ii) Incremental learning update fusion model After initial training, if updates to the cross-modal fusion model cease, its adaptability to environmental changes and new risk patterns will gradually decline over time. However, full retraining is computationally expensive and time-consuming, making it unsuitable for real-time security scenarios. Therefore, using collected data, incremental learning is performed to update the parameters of the S2 multimodal fusion model. The model employs an online stochastic gradient descent algorithm, with the loss function defined as: ; in, These are model weight parameters. This represents the input feature vector (such as fused multimodal data). It is a label for the real environment (such as risk category). It is the cross-entropy loss function. Cross-entropy loss has good gradient response characteristics to classification bias and can effectively guide the model parameters to converge towards the correct classification boundary. This refers to the batch size. The parameter update rule is as follows: ; in, It is an adaptive learning rate. It is a time index. This represents gradient calculation. This incremental approach avoids full retraining, adapts to environmental changes in real time, improves model accuracy, and has the beneficial effects of significantly reducing computational overhead and enhancing the system's ability to generalize to new risks.

[0097] (III) Reinforcement learning to optimize scheduling strategies The quality of task allocation strategies directly determines the collaborative efficiency and global rejection effectiveness of heterogeneous robots. Fixed rules or static strategies cannot adapt to dynamically changing risk situations and need to be continuously optimized through online learning. Therefore, reinforcement learning experience tuples are constructed, with each tuple being... ,in, It refers to the current overall situation status (such as risk distribution and robot location). It refers to scheduling actions (such as assigning robot types). It is an immediate reward, a denial of effectiveness, calculated as follows: ; and These are weighting coefficients. This reward function comprehensively considers the quality and timeliness of task execution, guiding the strategy to reduce response latency while ensuring a high rejection success rate. This is the next state. The experience tuples are input into the reinforcement learning framework, and the value table of the task assignment policy is updated using a temporal difference method. Q The learning update formula is: ; in, It is a state Action value function It's the learning rate. It is a discount factor (balancing future rewards). This represents the maximization operation. Iteratively updating the value table using a time-difference method allows the scheduling strategy to autonomously approach the optimum without pre-defined rules. The optimized multimodal fusion model parameters... and Strategy Value Table Through trigger-based mechanisms (such as when the model accuracy improves beyond a threshold) When the value of the strategy changes significantly or periodically (e.g., every 5 minutes), the data is sent to the edge robot using AES-256 encryption, completing a full closed loop from perception, decision-making, execution to evaluation and optimization, driving iterative improvement of the overall system performance.

[0098] The present invention also provides a heterogeneous multi-robot cooperative perception and rejection system, comprising: The edge perception and feature extraction module, deployed on each heterogeneous robot node, is used for: S1: Multiple heterogeneous robot nodes synchronously collect environmental data through their onboard multimodal sensors, preprocess and extract features from the collected data locally, generate compressed feature vectors representing the semantics of the environment, and upload them to the central platform. The central integration and situation building module, deployed on the central platform, is used for: S2, the central platform receives feature vectors and raw data uploaded by each heterogeneous robot node, performs cross-modal data association and fusion, generates unified multimodal fusion features and constructs a global situation map covering the monitoring area based on these features; at the same time, based on the identity of each robot, it obtains its capability indicators and quantifies them into structured capability vectors; The risk assessment and task allocation module, deployed on the central platform, is used for: S3, the central platform extracts high-level risk features based on the global situation map, performs risk level assessment and generates task requirements; calculates the matching degree between the capability vector of each robot and the task requirements, and performs attenuation correction based on the spatial distance of the robot's current position relative to the task area to obtain the scheduling matching score, and dynamically allocates tasks with the goal of optimizing the global response time. The graded rejection execution module is deployed on each of the scheduled heterogeneous robot nodes and is used to: S4, the scheduled heterogeneous robot nodes continuously track the designated target, calculate the intent score based on the target's kinematic behavior and posture characteristics, and generate a local risk score by combining the behavior anomaly degree; initiate a hierarchical rejection response based on the local risk score, and initiate the corresponding level of rejection measures when the local risk score and the global risk assessment value issued by the central platform reach the preset conditions. The online learning and strategy optimization module, deployed on the central platform, is used for: S5, the central platform continuously collects feedback data on perception accuracy and rejection effectiveness, incrementally updates the parameters of the cross-modal fusion model, and uses reinforcement learning based on experience tuples consisting of situational states, scheduling actions, and effectiveness rewards. The approach continuously optimizes task allocation strategies, forming an iterative closed loop of perception, decision-making, execution, and optimization.

[0099] The above description is merely an embodiment and does not constitute any limitation on the present invention. Any person skilled in the art can make many possible variations, modifications, or alterations to the technical solutions of the present invention without departing from the scope of the present invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention, without departing from the scope of the present invention, should fall within the protection scope of the present invention.

Claims

1. A heterogeneous multi-robot cooperative perception and denial method, characterized in that, Includes the following steps: S1: Multiple heterogeneous robot nodes synchronously collect environmental data through their onboard multimodal sensors, preprocess and extract features from the collected data locally, generate compressed feature vectors representing the semantics of the environment, and upload them to the central platform. S2, the central platform receives feature vectors and raw data uploaded by each heterogeneous robot node, performs cross-modal data association and fusion, generates unified multimodal fusion features and constructs a global situation map covering the monitoring area based on these features; at the same time, based on the identity of each robot, it obtains its capability indicators and quantifies them into structured capability vectors; S3, the central platform extracts high-level risk characteristics based on the global situation map, performs risk level assessment, and generates task requirements; The matching degree between the capability vector of each robot and the task requirements is calculated, and the attenuation correction is made by combining the spatial distance of the robot's current position relative to the task area to obtain the scheduling matching score. The task is dynamically allocated with the goal of optimizing the global response time. S4, the scheduled heterogeneous robot nodes continuously track the designated target, calculate the intent score based on the target's kinematic behavior and posture characteristics, and generate a local risk score by combining the behavior anomaly degree; initiate a hierarchical rejection response based on the local risk score, and initiate the corresponding level of rejection measures when the local risk score and the global risk assessment value issued by the central platform reach the preset conditions. S5, the central platform continuously collects feedback data on perception accuracy and rejection effectiveness, updates the parameters of the cross-modal fusion model incrementally, and continuously optimizes the task allocation strategy through reinforcement learning based on the experience tuple consisting of situational state, scheduling actions and effectiveness rewards, forming an iterative closed loop of perception, decision-making, execution and optimization.

2. The method of claim 1, wherein, In step S1, the synchronous acquisition of environmental data includes: providing a local time reference through a hardware clock, correcting deviations using a network time protocol, and achieving microsecond-level timestamp alignment between data streams from various modal sensors; The preprocessing includes: applying Gaussian filtering to the visible light image, applying median filtering to the infrared data, removing statistical outliers from the lidar point cloud based on Mahalanobis distance, and normalizing the data for each modality. The feature extraction is achieved through a lightweight convolutional neural network deployed locally on the robot.

3. The method according to claim 1, characterized in that, In step S2, the cross-modal data association and fusion adopts an encoder based on a multi-head attention mechanism. After mapping the features of different modalities to a shared dimension, the multi-modal fusion features are generated by calculating attention weights and performing weighted fusion. The global situation map is constructed using a graph neural network, which discretizes the monitoring area into grid nodes. The multimodal fusion features are used as the initial features of the nodes, and the node status is updated through a message passing mechanism based on mean pooling.

4. The method according to claim 1, characterized in that, In step S2, obtaining the capability indicators of each robot based on its identity identifier and quantifying them into a structured capability vector includes: The robot's unique device identifier is mapped to the capability database using a hash function. At least one performance indicator is queried and extracted, including drone endurance and hovering accuracy, wheeled robot maximum speed and obstacle crossing ability. After normalizing each indicator, they are concatenated into a fixed-dimensional structured capability vector.

5. The method according to claim 1, characterized in that, In step S3, the extraction of high-level risk features includes: using a density-based spatial clustering algorithm to identify the abnormal clustering features of the target, and using the difference method to calculate the instantaneous velocity and acceleration of the target between consecutive frames to identify high-speed maneuver trajectory features; The risk level assessment uses a logistic regression model to classify risk levels.

6. The method according to claim 1, characterized in that, In step S3, the matching degree calculation is specifically as follows: the robot's capability vector and the task requirement vector are multiplied by an inner product to obtain the basic matching degree, and the basic matching degree is multiplied by an exponential decay factor based on the distance from the robot's current position to the center of the task area to obtain the scheduling matching score. The target dynamic task allocation adopts a greedy algorithm, iteratively selecting the robot-task pair with the highest scheduling matching score. When the number of available robots is not less than the number of risks to be processed, one-to-one allocation is performed; when the number is insufficient, a multi-robot collaboration mode is activated.

7. The method according to claim 1, characterized in that, In step S4, the calculation of the intent score based on the kinematic behavior and posture features of the target includes: calculating the rate of change of velocity through the spatial coordinates of the target in consecutive frames, determining the angle between the target's posture orientation and the protected area, and performing a weighted linear combination of the rate of change of velocity and the cosine value of the angle to obtain a composite intent score. The local risk score is generated by weighted fusion of the composite intent score and the behavioral anomaly degree based on the deviation statistics of movement patterns; the fusion of the local risk score and the global risk assessment value adopts an equal weighted calculation to calculate the joint assessment value.

8. The method according to claim 1, characterized in that, In step S4, the initiation of the graded rejection response includes: When the local risk score exceeds the first threshold, an audible and visual warning is triggered. When the second threshold is exceeded, the probability of non-lethal denial measures is corrected by using an exponential decay model in combination with the ambient wind speed, and the execution of denial actions is controlled. When the joint assessment value obtained by fusing the local risk score and the global risk assessment value exceeds the highest rejection threshold, the impact force is calculated based on the mass of the rejection load and the target predicted acceleration, and kinetic rejection is initiated. After execution, the rejection effectiveness is assessed based on the ratio of the actual risk reduction to the expected reduction.

9. The method according to claim 1, characterized in that, In step S5, the cross-modal fusion model parameters are updated incrementally using an online stochastic gradient descent algorithm with cross-entropy as the loss function, and the collected feedback data on perception accuracy and rejection effectiveness are used to iteratively update the model parameters. The reinforcement learning method uses a time difference approach to update the value table of the task allocation strategy. The experience tuple consists of the current global situation state, the executed scheduling action, the next global situation state, and the immediate reward. The immediate reward is calculated based on a weighted combination of rejection success rate and response latency.

10. A heterogeneous multi-robot cooperative sensing and rejection system, characterized in that, include: The edge perception and feature extraction module, deployed on each heterogeneous robot node, is used for: S1: Multiple heterogeneous robot nodes synchronously collect environmental data through their onboard multimodal sensors, preprocess and extract features from the collected data locally, generate compressed feature vectors representing the semantics of the environment, and upload them to the central platform. The central integration and situation building module, deployed on the central platform, is used for: S2, the central platform receives feature vectors and raw data uploaded by each heterogeneous robot node, performs cross-modal data association and fusion, generates unified multimodal fusion features and constructs a global situation map covering the monitoring area based on these features; at the same time, based on the identity of each robot, it obtains its capability indicators and quantifies them into structured capability vectors; The risk assessment and task allocation module, deployed on the central platform, is used for: S3, the central platform extracts high-level risk characteristics based on the global situation map, performs risk level assessment, and generates task requirements; The matching degree between the capability vector of each robot and the task requirements is calculated, and the attenuation correction is made by combining the spatial distance of the robot's current position relative to the task area to obtain the scheduling matching score. The task is dynamically allocated with the goal of optimizing the global response time. The graded rejection execution module is deployed on each of the scheduled heterogeneous robot nodes and is used to: S4, the scheduled heterogeneous robot nodes continuously track the designated target, calculate the intent score based on the target's kinematic behavior and posture characteristics, and generate a local risk score by combining the behavior anomaly degree; initiate a hierarchical rejection response based on the local risk score, and initiate the corresponding level of rejection measures when the local risk score and the global risk assessment value issued by the central platform reach the preset conditions. The online learning and strategy optimization module, deployed on the central platform, is used for: S5, the central platform continuously collects feedback data on perception accuracy and rejection effectiveness, updates the parameters of the cross-modal fusion model incrementally, and continuously optimizes the task allocation strategy through reinforcement learning based on the experience tuple consisting of situational state, scheduling actions and effectiveness rewards, forming an iterative closed loop of perception, decision-making, execution and optimization.

Images