Abnormal behavior identification method and device of multi-agent system, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, the technical scheme can be applied to the field of industrial automation, the field of intelligent transportation and the field of intelligent power grids, and discloses an abnormal behavior recognition method, device and equipment of a multi-agent system and a medium. Meanwhile, the abnormal behavior score is calculated in parallel by the abnormal detection module; and after abnormity is judged through a dynamic threshold value, a closed-loop correction mechanism is triggered, and a reward function and an experience sampling weight of a strategy module are adjusted in real time, so that online compensation and optimization are performed on the behaviors of the intelligent agent. According to the method, the anomaly recognition accuracy, the real-time response capability and the overall system robustness of the multi-agent system in a complex dynamic environment are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, industrial automation, intelligent transportation, and smart grids, and particularly to a method, apparatus, device, and medium for identifying abnormal behavior in a multi-agent system. Background Technology

[0002] With the rapid development of automation and intelligent technologies, multi-agent systems are increasingly being applied in fields such as industrial manufacturing, intelligent transportation, and distributed energy management. These systems, through the collaborative cooperation of multiple agents, can accomplish complex tasks and improve overall efficiency. However, the increasing scale of the system and the growing complexity of the environment significantly increase the risk of agents exhibiting abnormal behavior due to hardware failures, communication interference, algorithmic defects, or sudden environmental changes. How to identify and handle these abnormal behaviors in a timely and accurate manner, ensuring the stable and reliable operation of the entire system, has become a key technical challenge that urgently needs to be addressed in this field.

[0003] Currently, traditional methods for detecting abnormal behavior in the industry have many limitations. Methods based on pre-defined rule bases rely on expert experience to define static rules. While effective in simple scenarios, they cannot adapt to dynamic, unstructured environments, and rule maintenance is costly and inflexible. Methods using single machine learning algorithms such as Isolation Forest and Support Vector Machine typically only analyze static snapshot data (such as position and velocity) of the agent offline, failing to correlate it with the agent's decision-making process. This results in an inability to distinguish between strategic behavior adjustments and genuine anomalies, leading to high false alarm rates and a lack of real-time response capabilities. Although some research utilizes deep reinforcement learning to optimize the agent's cooperative strategy, its focus is on improving policy performance and it lacks an inherent anomaly monitoring and handling mechanism. When an agent exhibits an anomaly, the system cannot autonomously perceive and repair it, still requiring external human intervention, causing response delays and potentially leading to serious consequences.

[0004] In summary, existing technical solutions generally treat policy optimization and anomaly detection as two independent processes, lacking an effective closed-loop linkage mechanism. This makes it difficult for systems to achieve high-precision, low-latency autonomous anomaly identification and recovery in complex dynamic environments, constituting a major obstacle for multi-agent systems to achieve high autonomy. Therefore, there is an urgent need for an integrated solution that deeply combines the decision-making advantages of reinforcement learning with the monitoring capabilities of anomaly detection. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for identifying abnormal behavior in multi-agent systems, aiming to improve the accuracy of anomaly identification in complex dynamic environments.

[0006] Firstly, a method for identifying abnormal behavior in a multi-agent system is provided, comprising the following steps: Real-time acquisition of state data of each agent in a multi-agent system; The state data is input into a pre-trained policy generation module to generate action instructions for each agent; the policy generation module is built based on a deep Q-network and is trained through a priority experience replay mechanism to realize the mapping from state to action. The status data and the action instructions are input into the anomaly detection module for real-time analysis to generate an anomaly score. The abnormal score is compared with a dynamic threshold. If it is determined to be abnormal, a policy correction signal is triggered. In response to the policy correction signal, a correction mechanism is initiated to correct the action instructions generated by the policy generation module; The revised action instructions are sent to the corresponding agents to control their execution.

[0007] Secondly, an abnormal behavior recognition device for a multi-agent system is provided, comprising: The state data acquisition module is used to acquire the state data of each agent in the multi-agent system in real time. An action instruction generation module is used to input the state data into a pre-trained policy generation module to generate action instructions for each agent; the policy generation module is built based on a deep Q-network and is trained through a priority experience replay mechanism to realize the mapping from state to action. An anomaly score generation module is used to input the status data and the action command into the anomaly detection module for real-time analysis and to generate an anomaly score. The correction signal triggering module is used to compare the abnormal score with the dynamic threshold. If it is determined to be abnormal, the strategy correction signal is triggered. An action command correction module is used to respond to the strategy correction signal by activating a correction mechanism to correct the action commands generated by the strategy generation module. The action command issuing module is used to issue the modified action commands to the corresponding intelligent agents to control the execution of the intelligent agents.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for identifying abnormal behavior of a multi-agent system.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described method for identifying abnormal behavior in a multi-agent system.

[0010] In the above-mentioned scheme for identifying abnormal behavior in a multi-agent system, the state data of each agent in the multi-agent system is acquired in real time; the state data is input into a pre-trained policy generation module to generate action instructions for each agent; the policy generation module is built based on a deep Q-network and trained through a priority experience replay mechanism to realize the mapping from state to action; the state data and the action instructions are input into an anomaly detection module for real-time analysis to generate an anomaly score; the anomaly score is compared with a dynamic threshold, and if it is determined to be abnormal, a policy correction signal is triggered; in response to the policy correction signal, a correction mechanism is initiated to correct the action instructions generated by the policy generation module; the corrected action instructions are sent to the corresponding agent to control the agent to execute. By deeply coupling the pre-trained reinforcement learning policy generation module with the real-time anomaly detection module, a closed-loop control system of perception-decision-monitoring-correction is constructed. This significantly improves the accuracy and real-time performance of abnormal behavior recognition, effectively distinguishes between strategic behavior and real anomalies, and reduces the false alarm rate. It also enables autonomous online compensation and policy adjustment for abnormal behavior, greatly reducing system response latency and dependence on manual intervention. Furthermore, it enhances the overall robustness and adaptability of the multi-agent system in dynamic and complex environments, ensuring the continuous and stable execution of collaborative tasks. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic flowchart of an abnormal behavior recognition method for a multi-agent system according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the process of generating action instructions in one embodiment of the present invention; Figure 3 This is a schematic diagram of a process for generating abnormal scores in one embodiment of the present invention; Figure 4 This is a schematic diagram of a process for modifying action commands in one embodiment of the present invention; Figure 5 This is a schematic diagram of an abnormal behavior recognition device for a multi-agent system in one embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 7This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Please see Figure 1 As shown, Figure 1 A flowchart illustrating an abnormal behavior recognition method for a multi-agent system provided in an embodiment of the present invention includes the following steps: S10. Real-time acquisition of state data of each agent in the multi-agent system; In this embodiment, in a multi-robot collaborative intelligent warehousing scenario, each AGV (Automated Guided Vehicle) intelligent agent periodically (e.g., at a frequency of 100Hz) collects its own state information through its own LiDAR, IMU (Inertial Measurement Unit), encoder, and UWB (Ultra-Wideband) positioning tag and other sensors. This raw state information constitutes the source for subsequent processing. The state data is a multi-dimensional vector, and its specific composition can be flexibly adjusted according to the application scenario. In a preferred AGV scheduling embodiment, it includes at least: the agent's X and Y coordinates in the world coordinate system (2D), the linear and angular velocities calculated by its drive wheel encoders (2D), the progress of its current task on the shelf it is carrying (1D, represented as the ratio of completed distance to total distance, normalized to the [0,1] interval), its Euclidean distance to the current target shelf point (1D), the distance to the three nearest cooperating AGVs obtained by UWB ranging (3D), the current signal reception strength indication (RSSI, 1D) reported by its Wi-Fi or 5G communication module, the remaining battery percentage provided by its battery management system (BMS) (1D), and a historical action success rate maintained by the agent itself (1D, e.g., the percentage of times a movement command was successfully executed in the last 100 attempts). These 12 primitive features collectively characterize the agent's comprehensive state at a specific moment, providing comprehensive perceptual input for subsequent intelligent decision-making.

[0015] S20. Input the state data into the pre-trained policy generation module to generate action instructions for each agent; the policy generation module is built based on a deep Q-network and is trained through a priority experience replay mechanism to realize the mapping from state to action. Furthermore, in one specific embodiment, step S20 specifically includes: S21. Standardize and preprocess the original state data of each agent in the multi-agent system to generate a standardized state vector. In this embodiment, due to the significant differences in the dimensions and numerical ranges of the state data in each dimension (for example, coordinates might be on the order of meters, while electricity is expressed as a percentage), directly inputting them into the neural network would lead to difficulties in model training and slow convergence. Therefore, standardization is necessary. This scheme employs the Z-score standardization method to process each feature dimension independently. Specifically, the system maintains a sliding window of 1000 steps and calculates the mean μ and standard deviation σ of each feature dimension within the window in real time. Newly acquired original feature values ​​x are standardized to obtain standardized feature values ​​with a mean of 0 and a standard deviation of 1. This dynamically updated standardization method can adapt to the slow drift of data distribution in the environment, avoiding the bias that may be introduced by fixed-parameter standardization, thereby ensuring the stability and consistency of the input data to the policy generation module and improving the generalization performance of the model at different running stages.

[0016] S22. Input the standardized state vector into the policy generation module, and generate action instructions for each agent based on the Q-value maximization principle.

[0017] In this embodiment, the policy generation module is a Deep Q-Network (DQN) that has undergone offline training and online fine-tuning. One specific implementation of its network structure is as follows: the input layer receives the aforementioned 12-dimensional normalized state vector; followed by three fully connected hidden layers with 256, 128, and 64 neurons respectively, all employing the ReLU activation function to introduce nonlinear transformation capabilities; the output layer corresponds one-to-one with all possible discrete actions of the agent (e.g., forward, backward, turn 30 degrees left, turn 30 degrees right, stop, etc.), outputting the Q-value (expected cumulative reward) for each action. Its workflow is as follows: during the application phase, the network performs forward propagation on the input state vector, calculating the Q-values ​​of all actions. The agent typically selects the action with the highest Q-value, but may also randomly select other actions with a small probability ε to discover potentially better strategies. In this embodiment, by employing a dynamically decaying ε-greedy strategy (e.g., ε starts at 0.9 and exponentially decays to 0.1 with each training step), a good balance between sufficient initial exploration and stable utilization in the later stages is achieved. Ultimately, the output action commands are transmitted wirelessly to the AGV's onboard controller for execution. The core function of this module is to map a high-dimensional, continuous state space to an optimal discrete action space, achieving end-to-end decision optimization.

[0018] S30. Input the status data and the action command into the anomaly detection module for real-time analysis to generate an anomaly score. Furthermore, in one specific embodiment, step S30 specifically includes: S31. Based on the state data and action instructions, extract a temporal feature vector; the temporal feature vector includes at least one of action sequence features, Q-value change features, and policy entropy features. In this embodiment, the anomaly detection module does not directly use the original state, but extracts temporal features that better reveal abnormal behavioral patterns. A specific feature engineering implementation is as follows: Action sequence features (3D): The actions performed by the agent in the five most recent consecutive time steps (e.g., forward, left turn, forward, stop, right turn) are treated as a sentence. A pre-trained Word2Vec word embedding model maps each discrete action to a dense 3D vector. The average of these five vectors or the last vector is taken to represent the embedded representation of the short-term action intent. This transforms discrete, symbolic action sequences into continuous, semantically meaningful numerical vectors, facilitating the model's capture of subtle behavioral pattern changes.

[0019] Q-value variation feature (1D): Calculate the first-order difference between the maximum Q-value of the policy network output at the current time step and the maximum Q-value at the previous time step. This feature effectively reflects the change in the agent's confidence in its own decisions. An abnormal, unexpected state transition may cause drastic fluctuations in the Q-value.

[0020] Policy entropy feature (1D): Calculates the Shannon entropy of the probability distribution of all actions in the output layer of the policy network. High entropy indicates that the agent is hesitant in the current state, and the estimated values ​​of multiple actions are similar; low entropy indicates that the agent is confident, and a certain action is significantly better than others. Abnormal states may cause the policy entropy to deviate from the normal pattern.

[0021] These carefully designed temporal features together form an 8-dimensional feature vector, which serves as the core input to the anomaly detection model, enabling it to deeply perceive behavioral anomalies at the decision-making level.

[0022] S32. Input the time-series feature vector into the detection model constructed based on the isolated forest algorithm to calculate the anomaly score.

[0023] In this embodiment, the anomaly detection model employs an improved Isolation Forest algorithm. Its specific operation mechanism is as follows: During the initialization phase, the system constructs 100 isolation trees (iTrees) using historical data from a normal operating period. Each tree recursively partitions the data space by randomly selecting features and split values. Anomalies, due to their different feature values ​​compared to the normal population, are generally easier to isolate, requiring only a shorter path length to be assigned to an isolated leaf node. During online detection, the newly arriving 8-dimensional temporal feature vector is sequentially passed through these 100 iTrees, and its path length in each tree is calculated and averaged. Finally, a normalization formula is used to convert the average path length into an anomaly score between 0 and 1. The closer the score is to 1, the greater the probability that the sample is an anomaly. This unsupervised detection method does not require pre-labeling of anomaly data, making it ideal for detecting unknown anomalies. It also boasts high computational efficiency, meeting real-time requirements.

[0024] S40. Compare the abnormal score with the dynamic threshold. If it is determined to be abnormal, trigger the strategy correction signal. Furthermore, in one specific embodiment, The step of comparing the anomaly score with a dynamic threshold and triggering a policy correction signal if an anomaly is determined includes: S41. Obtain a dynamic threshold, wherein the dynamic threshold is adaptively determined by calculating the statistical characteristics of abnormal scores within a window over a recent period; S42. Compare the anomaly score with the dynamic threshold; S43. If the abnormal score continues to exceed the dynamic threshold, the policy correction signal is triggered.

[0025] In this embodiment, using a fixed anomaly threshold cannot adapt to the natural changes in the normal behavior patterns of the system at different operating stages. Therefore, this solution designs a dynamic threshold adjustment mechanism. Specifically: The system maintains a sliding window of all anomaly scores calculated within the most recent 500 time steps. The dynamic threshold T is set as the mean score μ_S within the window plus three times the standard deviation σ_S. This means that only when a newly calculated anomaly score significantly deviates from the statistical distribution of recent normal behavior will it be judged as an anomaly. This adaptive mechanism effectively reduces the false alarm rate, enabling the system to distinguish between normal performance fluctuations and genuine abnormal behavior, thus improving the reliability of detection. When the anomaly score S(x) > T and persists for more than 3 consecutive time steps (anti-jitter), a policy correction signal is confidently triggered, and an anomaly level L between 0 and 1 is calculated based on the degree to which the threshold is exceeded.

[0026] S50. In response to the strategy correction signal, a correction mechanism is initiated to correct the action instructions generated by the strategy generation module; Furthermore, in one specific embodiment, step S50 specifically includes: S51. Determine the anomaly level based on the anomaly score; S52. Based on the anomaly level, apply a penalty to the action instructions generated by the strategy generation module to correct their reward function; S53. Adjust the experience sampling weights within the strategy generation module to make it more inclined to sample historical experiences similar to the current abnormal state.

[0027] In this embodiment, once an anomaly is confirmed, the system immediately switches from normal operation mode to correction mode. Its core correction mechanism includes two aspects: Real-time optimization of the reward function: In addition to the existing task reward (such as a positive reward for successfully reaching the target point), a negative reward (penalty) proportional to the anomaly level L is immediately added. This new reward signal immediately affects the next update of the policy network, making it aware that the current state or action is undesirable, thus prompting the network to quickly adjust its policy and output action instructions that avoid the current abnormal state. This is equivalent to injecting a sense of pain into the agent, enabling it to learn immediately from erroneous behavior.

[0028] Experience replay priority rescheduling: Simultaneously, the system suspends normal uniform sampling and instead quickly retrieves historical experiences from the experience replay pool that are most similar to the current anomalous state in the feature space (by calculating cosine similarity, retaining samples with sim > 0.7). The sampling priority of these samples is significantly increased (e.g., multiplied by a factor of (1 + 2L)). Thus, in subsequent network training updates, the model will focus on reviewing historical experiences related to the current predicament, accelerating the learning of how to cope with such anomalous situations. This targeted review greatly improves the efficiency and effectiveness of policy correction.

[0029] Furthermore, during the correction phase, the exploration rate ε is temporarily fixed at a relatively high level (e.g., 0.3) to encourage the agent to explore more in anomalous regions to find new and effective solutions, thus preventing the policy from getting stuck in local optima or solidifying anomalous behavior. This closed-loop correction mechanism achieves an automatic, rapid, and accurate closed loop from anomaly detection to policy adjustment.

[0030] S60. Send the revised action instructions to the corresponding agent to control the agent to execute them.

[0031] In this embodiment, the new action instructions generated after optimization by the strategy correction module are promptly sent to the malfunctioning AGV agent via a low-latency communication network (such as 5G URLLC). The agent executes the new instructions, for example, correcting its initial attempt to continue moving to cross a communication blind spot to first move 5 meters to the right and forward to find a stronger signal before continuing the task. This new action effectively avoids the risk of task failure due to communication interruption, ensuring the continuity and stability of the tasks of the individual agent and the entire system. The entire process, from detecting the anomaly to executing the correction instructions, is completed within milliseconds, achieving true online real-time compensation and greatly reducing reliance on manual monitoring and intervention.

[0032] Furthermore, in one specific embodiment, the abnormal behavior identification method for a multi-agent system further includes a multi-agent collaborative verification step, comprising: Calculate the behavioral consistency index between the current agent and its neighboring agents; If the behavior consistency index is lower than the preset consensus threshold, the frequency and accuracy of anomaly detection for the current agent will be increased.

[0033] In this embodiment, to overcome the limitations of single-agent perception, a distributed collaborative verification mechanism is introduced. Specifically, each agent periodically calculates its own behavioral consistency index with its k nearest neighbor agents (e.g., k=3) on a certain common feature (e.g., speed direction, task progress), and averages these indices to obtain its own consistency index. If the consistency index of an AGV consistently falls below a preset threshold (e.g., 0.3), it is marked as a high-risk node. Instead of arbitrarily determining an anomaly, the system initiates stricter monitoring: increasing the frequency of anomaly detection from 10Hz to 20Hz; and expanding the historical window used to extract temporal features from 5 steps to 10 steps. This approach allows for faster detection of anomalies, while the longer observation window provides richer contextual information to reduce the probability of false positives. This verification mechanism, based on group consensus, cleverly leverages the inherent advantages of multi-agent systems, transforming the unreliable perception of a single agent into reliable diagnosis by the group, significantly improving the robustness and accuracy of the entire system's anomaly identification.

[0034] Furthermore, in one specific embodiment, the abnormal behavior recognition method for a multi-agent system further includes a recovery determination step, comprising: After the policy correction signal is triggered, the abnormal score is continuously monitored; If the abnormal score falls below the dynamic threshold within multiple consecutive time steps, the correction mechanism is automatically deactivated. If the abnormal score continues to be higher than the dynamic threshold, a higher-level warning is triggered and a backup control strategy is activated.

[0035] In this embodiment, a robust recovery mechanism is crucial for ensuring the long-term stable operation of the system. After triggering the correction mode, the system continuously monitors changes in the anomaly score. A specific recovery judgment logic is as follows: if the anomaly score remains below the dynamic threshold T for 20 consecutive time steps, it indicates that the correction measures have taken effect and the agent's behavior has returned to normal. The system will automatically deactivate the correction mode, stop applying rewards and penalties and priority sampling, and the exploration rate will also return to the decay mode, allowing the entire system to smoothly switch back to normal optimized operation. Conversely, if the anomaly score remains high (e.g., still above T after more than 20 steps), it indicates that the current online adaptive correction cannot effectively handle the anomaly. At this time, the system will trigger a secondary warning, notify the background monitoring system to intervene, and automatically switch to a predefined and fully validated backup safety strategy (e.g., immediate stop, horn alarm, slow movement to the nearest safe zone, etc.) to ensure personal and equipment safety in the most conservative way and to buy time for manual intervention. This tiered recovery mechanism ensures that the system can self-heal in most cases and safely degrade in extreme cases, reflecting the meticulousness and robustness of the design.

[0036] The technical solution of this application can be applied to the fields of industrial automation, intelligent transportation and smart grid.

[0037] In the field of industrial automation, consider a multi-robot workstation that collaboratively assembles large components (such as car bodies). This workstation consists of several six-axis industrial robots, AGV material handling vehicles, and a central control system, forming a typical multi-agent collaborative scenario. Each robot agent is responsible for different processes, such as gripping, welding, gluing, or installation. Their work cycles must be highly synchronized, and their spatial paths must be precisely avoided. Even a minor anomaly in any single agent can cause the entire production line to stop, resulting in huge economic losses.

[0038] In this scenario, each robotic agent reports its state data in real time, which forms the input to the reinforcement learning strategy module in the patented method. The state data includes not only the robot's joint angles, the three-dimensional coordinates of the end effector, and linear and angular velocities, but also crucial task context information, such as the current workstation's task completion progress, distance to the target installation point, minimum distance to the collaborating robot, and readings from force and torque sensors to sense assembly resistance. All these multi-dimensional features, after standardization, are input to a strategy generation module that has been pre-trained extensively in a digital twin environment. This module is essentially a deep Q-network that has learned which actions (such as movement trajectory and welding parameters) should be performed in which states to complete the task most efficiently and safely. The module's output is the optimal action command for the robot's next time step.

[0039] However, real factory environments are fraught with uncertainty. A robot performing glue application might experience abnormal glue dispensing due to a slight blockage in the glue gun nozzle. Although its movement trajectory may not deviate significantly, the glue application quality will still be substandard. In such cases, traditional vision-based end-point inspection often only detects the defect after the process is completed, exhibiting a lag. The anomaly detection module in this patent, however, can provide real-time warnings during the process. This module continuously monitors the temporal behavioral characteristics of each robot: it not only analyzes the deviation between the glue application robot's own action sequence and the expected model, but also pays attention to changes in its Q-value. An optimal action that was initially confident (high Q-value) may have a lower actual reward due to execution obstruction (poor glue dispensing), reflected as abnormal fluctuations in the Q-value sequence. Simultaneously, the module also calculates the collaboration frequency between the robot and adjacent welding robots. If the interaction is interrupted due to its own anomaly, this will also be captured as a drop in the collaboration index.

[0040] When the anomaly detection module calculates an anomaly score based on these features and the score exceeds the dynamic threshold, the system does not immediately alarm and shut down. Instead, it first triggers the patented core closed-loop correction mechanism. The strategy correction mechanism is activated: the system immediately retrieves similar scenarios from the historical experience pool (e.g., past experience handling unstable dispensing due to slight changes in adhesive viscosity) and prioritizes sampling these experiences to quickly fine-tune the strategy network. Simultaneously, the reward function is optimized in real-time, applying a negative reward to continue the current low-quality dispensing action, prompting the strategy network to rapidly adjust its output and decide on corrective instructions such as slightly reducing the moving speed to increase dispensing pressure or executing a preset cleaning backup action. The entire process is completed within milliseconds.

[0041] More importantly, the system possesses multi-agent collaborative verification capabilities. If an AGV begins to slightly deviate from its path due to navigation signal drift, its own sensors may not immediately detect this anomaly. However, adjacent stationary robots will calculate the difference between their real-time distance and expected distance to the AGV, identifying a significant decrease in behavioral consistency. The system then marks the AGV as a high-risk node and doubles its anomaly detection frequency, thereby capturing the early signs of anomalies more accurately and earlier. Through policy correction instructions, the system guides the AGV to a safe stop or replans its path, avoiding potential collisions with personnel or other equipment.

[0042] Through the application of this patented technology, the intelligent assembly line has achieved a fundamental transformation from passively responding to alarms to proactive predictive maintenance. The system can absorb environmental uncertainties, transforming many anomalies that previously required line shutdown into events that are autonomously resolved online. This significantly improves the overall equipment efficiency (OEE) and flexible production capabilities of the production line, providing crucial technical support for a truly "lights-out" factory.

[0043] In the field of intelligent transportation, this patented technology can be perfectly applied to the cooperative cruise control of connected autonomous driving fleets. The fleet consists of multiple autonomous trucks with V2V (vehicle-to-vehicle) communication capabilities, traveling in platoons with extremely close spacing to reduce wind resistance, save energy, and increase road throughput. In this high-speed, complex, and dynamic system, any abnormal behavior of any vehicle could trigger a chain reaction, leading to a serious accident.

[0044] In this scheme, each vehicle in the convoy is an intelligent agent. They continuously share their own state, including latitude and longitude coordinates, speed, acceleration, heading angle, distance to vehicles in front and behind, battery level (if electric vehicles), and remaining range of the planned route, through onboard sensors and V2V communication. This data is aggregated as input to a policy generation module located locally or at the edge cloud. This module is a deep reinforcement learning network trained on massive amounts of urban and highway road conditions. Its core responsibility is to output energy-efficient, safe, and comfortable longitudinal and lateral control commands to maintain the convoy's stable formation and smooth driving.

[0045] The anomaly detection module constantly monitors for any deviations from the cooperative driving strategy. Its analysis of temporal characteristics is highly traffic-specific: for example, it continuously monitors the strategy entropy of the lead vehicle's decision-making actions. Under normal circumstances, the lead vehicle's decisions regarding the road conditions ahead should be decisive and clear (low entropy); if its decisions become hesitant (high entropy), it may indicate that its sensors (such as lidar) have detected obstacles that are difficult to identify. Simultaneously, the module calculates the first-order difference (ΔQ) of each vehicle's Q-value. A vehicle following normally should have a relatively stable Q-value; if a following vehicle's camera is suddenly obstructed by dirt, causing a decrease in perception confidence, its Q-value will fluctuate drastically, a strong anomaly signal. Furthermore, the frequency of interaction between vehicles and their neighbors (vehicles in front and behind) is also a key indicator; once the communication link becomes unstable or latency increases, cooperative consistency will immediately decrease.

[0046] When a vehicle's score exceeds the limit due to a sudden sensor malfunction, the system instantly triggers closed-loop control. The strategy correction mechanism intervenes immediately: instead of abruptly forcing the vehicle to brake, it first optimizes the reward function by adding a strong constraint penalty term for maintaining a safe distance to the original reward objective of keeping close to the vehicle in front. This forces the policy network to immediately output new instructions: for example, smoothly reduce its own speed while broadcasting a deceleration signal to vehicles behind via V2V communication, guiding the following convoy to synchronously and smoothly increase the distance between vehicles, creating a safe buffer space. The behavior of the entire convoy seamlessly transitions from a close cooperative mode to a safe defensive mode.

[0047] Simultaneously, the multi-agent verification mechanism begins operation. Once the lead vehicle detects a sudden drop in the consistency index of an abnormal behavior in a vehicle within the convoy, the system immediately marks it as a high-risk vehicle and upgrades its monitoring level to 20Hz. This helps the system distinguish in the first instance whether the vehicle is performing a reasonable emergency obstacle avoidance maneuver (such as avoiding a vehicle suddenly changing lanes) or has truly fallen into an uncontrollable state beyond its autonomous handling capabilities. If it is the latter, the system will decisively trigger a level-two warning, instructing the vehicle to execute the ultimate backup strategy of pulling over, and guiding the entire convoy to regroup and continue its journey, thereby maximizing the safety and smoothness of traffic flow. This patented technology transforms autonomous driving convoys from fragile behavioral replicas into resilient, self-diagnostic, and collectively recovering intelligent entities.

[0048] In the field of smart grids, smart grids face the challenge of significant volatility and uncertainty brought about by the high proportion of renewable energy integration. This patented technology can be applied to regional distributed energy management systems (DEMS), treating each distributed energy source (such as photovoltaic power plants, wind turbines, energy storage battery packs, and even dispatchable industrial loads) within the region as an intelligent entity. They work together to complete the macro-level tasks of peak shaving and valley filling, balancing fluctuations, and maintaining grid stability.

[0049] Each energy agent continuously reports its status data, including power generation / consumption, voltage, current, frequency, battery SOC (state of charge), and predicted power generation / consumption curves. This data is input into a regional-level strategy generation module, which is a virtual power plant dispatching brain trained with reinforcement learning. The core objective of this module is to output optimal dispatching instructions (such as increasing generation, reducing generation, charging, and discharging) to maximize economic benefits and minimize carbon emissions for the entire region while satisfying grid dispatching instructions.

[0050] The anomaly detection module acts as the power grid's immune system, constantly identifying various anomalies. Its analysis of temporal characteristics closely aligns with power properties: it monitors the sequence of actions related to the output power of photovoltaic power plants to see if it matches the expected instructions generated by the strategy. If the instruction is to increase output, but the actual power continues to decline, an anomaly may exist. It calculates state transition entropy to determine if the operating state of local nodes in the power grid is on the verge of unpredictable chaos. More importantly, it monitors the deviation of each agent from the optimal action. If an energy storage unit refuses to execute the optimal instruction to discharge during peak electricity price periods, its deviation will increase sharply, potentially indicating an internal fault or communication interruption.

[0051] When a sudden cloud cover causes a photovoltaic power station's output to plummet within minutes, its anomaly score quickly exceeds the limit. This patent's closed-loop correction mechanism responds swiftly. The strategy correction mechanism is activated: the system immediately introduces a penalty term for the power deficit into the reward function. This negative reward drives the strategy generation module to quickly recalculate and issue new scheduling instructions: this might involve instructing the standby gas turbine to start rapidly, instructing the energy storage battery to discharge immediately, or sending signals to certain interruptible loads requesting reduced power consumption. This process is dynamic and rolling until power is restored to balance.

[0052] The multi-agent verification mechanism plays a crucial role here. A sudden, abnormal increase in load reported by a smart meter in a residential area could be normal electricity consumption or a device malfunction. The system determines the authenticity by calculating the consistency of this increase with the behavior of neighboring transformers and distributed power sources. If only this single point is abnormal without any associated changes in the surrounding area, it is confirmed as an anomaly; if it occurs simultaneously with a decrease in photovoltaic output, it is likely a genuine demand for electricity. Through this collaborative verification, the system significantly reduces false alarms, ensures the accuracy of dispatch commands, avoids unnecessary load shedding, guarantees power supply reliability, and enables the power grid to maintain safe and stable operation while absorbing more green energy.

[0053] like Figure 6 As shown, this embodiment of the invention also provides an abnormal behavior recognition device for a multi-agent system, comprising: The state data acquisition module 10 is used to acquire the state data of each agent in the multi-agent system in real time. Action instruction generation module 20 is used to input the state data into a pre-trained policy generation module to generate action instructions for each agent; the policy generation module is built based on a deep Q network and is trained through a priority experience replay mechanism to realize the mapping from state to action. Anomaly score generation module 30 is used to input the status data and the action command into the anomaly detection module for real-time analysis to generate anomaly score; The correction signal triggering module 40 is used to compare the abnormal score with the dynamic threshold, and if it is determined to be abnormal, it triggers the strategy correction signal. Action command correction module 50 is used to respond to the strategy correction signal and activate the correction mechanism to correct the action commands generated by the strategy generation module; The action command issuing module 60 is used to issue the modified action command to the corresponding intelligent agent to control the intelligent agent to execute.

[0054] In one specific embodiment, the action instruction generation module 20 is specifically used for: The state data of each agent in the acquired multi-agent system are standardized and preprocessed to generate standardized state vectors. The standardized state vector is input into the policy generation module, and action instructions for each agent are generated based on the Q-value maximization principle.

[0055] In one specific embodiment, the abnormal score generation module 30 is specifically used for: Based on the state data and action instructions, a temporal feature vector is extracted; the temporal feature vector includes at least one of action sequence features, Q-value change features, and policy entropy features. The temporal feature vector is input into the detection model constructed based on the isolated forest algorithm to calculate the anomaly score.

[0056] In the correction signal triggering module 40, the dynamic threshold is adaptively determined by calculating the statistical characteristics of the abnormal scores within a window over a recent period.

[0057] In one specific embodiment, the action command correction module 50 is specifically used for: The level of abnormality is determined based on the abnormality score. Based on the anomaly level, a penalty term is applied to the action instructions generated by the strategy generation module to correct their reward function; Adjust the experience sampling weights within the strategy generation module to make it more inclined to sample historical experiences similar to the current abnormal state.

[0058] Furthermore, in one specific embodiment, the abnormal behavior recognition device for a multi-agent system further includes a multi-agent collaborative verification module, specifically used for: Calculate the behavioral consistency index between the current agent and its neighboring agents; If the behavior consistency index is lower than the preset consensus threshold, the frequency and accuracy of anomaly detection for the current agent will be increased.

[0059] Furthermore, in one specific embodiment, the abnormal behavior recognition device for a multi-agent system further includes a recovery determination module, specifically used for: After the policy correction signal is triggered, the abnormal score is continuously monitored; If the abnormal score falls below the dynamic threshold within multiple consecutive time steps, the correction mechanism is automatically deactivated. If the abnormal score continues to be higher than the dynamic threshold, a higher-level warning is triggered and a backup control strategy is activated.

[0060] Specific limitations regarding the abnormal behavior recognition device for multi-agent systems can be found in the limitations of the abnormal behavior recognition method for multi-agent systems described above, and will not be repeated here. Each module in the aforementioned abnormal behavior recognition device for multi-agent systems can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0061] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the server-side functions or steps of an abnormal behavior recognition method for a multi-agent system.

[0062] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the client-side functions or steps of an abnormal behavior recognition method for a multi-agent system.

[0063] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: S10. Real-time acquisition of state data of each agent in the multi-agent system; S20. Input the state data into the pre-trained policy generation module to generate action instructions for each agent; the policy generation module is built based on a deep Q-network and is trained through a priority experience replay mechanism to realize the mapping from state to action. S30. Input the status data and the action command into the anomaly detection module for real-time analysis to generate an anomaly score. S40. Compare the abnormal score with the dynamic threshold. If it is determined to be abnormal, trigger the strategy correction signal. S50. In response to the strategy correction signal, a correction mechanism is initiated to correct the action instructions generated by the strategy generation module; S60. Send the revised action instructions to the corresponding agent to control the agent to execute them.

[0064] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: S10. Real-time acquisition of state data of each agent in the multi-agent system; S20. Input the state data into the pre-trained policy generation module to generate action instructions for each agent; the policy generation module is built based on a deep Q-network and is trained through a priority experience replay mechanism to realize the mapping from state to action. S30. Input the status data and the action command into the anomaly detection module for real-time analysis to generate an anomaly score. S40. Compare the abnormal score with the dynamic threshold. If it is determined to be abnormal, trigger the strategy correction signal. S50. In response to the strategy correction signal, a correction mechanism is initiated to correct the action instructions generated by the strategy generation module; S60. Send the revised action instructions to the corresponding agent to control the agent to execute them.

[0065] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0066] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0067] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0068] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for identifying abnormal behavior of a multi-agent system, characterized by, The method comprises the following steps: Real-time acquisition of state data of each agent in a multi-agent system; Inputting the state data into a pre-trained policy generation module to generate action instructions for each agent; the policy generation module is constructed based on a deep Q network and trained through a priority experience replay mechanism, and is used to realize the mapping from state to action; Inputting the state data and the action instructions into an anomaly detection module for real-time analysis to generate an anomaly score; Comparing the anomaly score with a dynamic threshold value, and if it is determined to be abnormal, triggering a policy correction signal; In response to the policy correction signal, starting a correction mechanism to correct the action instructions generated by the policy generation module; Downlinking the corrected action instructions to the corresponding agent to control the agent to execute. 2.The method of claim 1, wherein, The step of inputting the state data into the pre-trained policy generation module to generate action instructions for each agent comprises: Standardizing and preprocessing the acquired state data of each agent in the multi-agent system to generate a standardized state vector; Inputting the standardized state vector into the policy generation module to generate action instructions for each agent based on the principle of maximizing Q value. 3.The method of claim 1, wherein, The step of inputting the state data and the action instructions into the anomaly detection module for real-time analysis to generate an anomaly score comprises: Based on the state data and the action instructions, extracting a time sequence feature vector; the time sequence feature vector comprises at least one of action sequence features, Q value change features, and policy entropy features; Inputting the time sequence feature vector into a detection model constructed based on an isolation forest algorithm to calculate the anomaly score. 4.The method of claim 1, wherein, The step of comparing the anomaly score with a dynamic threshold value, and if it is determined to be abnormal, triggering a policy correction signal comprises: Obtaining a dynamic threshold value, which is adaptively determined by calculating statistical features of anomaly scores in a recent time window; Comparing the anomaly score with the dynamic threshold value; If the anomaly score continuously exceeds the dynamic threshold value, triggering the policy correction signal. 5.The method of claim 1, wherein, The step of responding to the policy correction signal and starting a correction mechanism to correct the action instructions generated by the policy generation module comprises: Determining an anomaly level according to the anomaly score; According to the anomaly level, imposing a penalty term on the action instructions generated by the policy generation module to correct its reward function; Adjusting the experience sampling weight inside the policy generation module to make it tend to sample historical experiences similar to the current abnormal state. 6.The method of claim 1, wherein, Further comprising a multi-agent collaborative verification step, comprising: Calculating a behavior consistency index between the current agent and the remaining neighbor agents; If the behavior consistency index is lower than a preset consensus threshold, increasing the anomaly detection frequency and detection accuracy of the current agent. 7.The method of claim 1, wherein, Further comprising a recovery determination step, comprising: After triggering the policy correction signal, continuously monitoring the anomaly score; If the anomaly score falls below the dynamic threshold value for a plurality of consecutive time steps, automatically releasing the correction mechanism; If the anomaly score continuously exceeds the dynamic threshold value, triggering a higher level of early warning and enabling a backup control strategy.

8. An abnormal behavior recognition apparatus of a multi-agent system, characterized by comprising: Comprise: The state data acquisition module is configured to acquire state data of each agent in the multi-agent system in real time. The action instruction generation module is configured to input the state data into a pre-trained policy generation module to generate an action instruction of each agent. The abnormal score generation module is configured to input the state data and the action instruction into an anomaly detection module to generate an abnormal score in real time. The correction signal triggering module is configured to compare the abnormal score with a dynamic threshold, and trigger a policy correction signal if the abnormal score is determined to be abnormal. The action instruction correction module is configured to start a correction mechanism to correct the action instruction generated by the policy generation module in response to the policy correction signal. The action instruction issuing module is configured to issue the corrected action instruction to the corresponding agent to control the agent to execute.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the abnormal behavior identification method of the multi-agent system according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the abnormal behavior identification method of the multi-agent system according to any one of claims 1 to 7.

Images