Vulnerability intelligent collaborative detection method and system based on multi-agent reinforcement learning

Abstract

The invention discloses a vulnerability intelligent collaborative detection method and system based on multi-agent reinforcement learning. The method comprises the following steps: independently executing detection according to local observation information and a strategy generation model through an intelligent agent deployed at an edge node; the strategy generation model is constructed based on a decentralized partial observable Markov decision process, and is obtained through cloud centralized training and global shared reward function optimization. When the agent obtains a forward reward, packaging a corresponding vulnerability context and related strategy parameters into an experience package; and the experience packet is diffused in the cooperative detection network in a mode of randomly selecting neighbor nodes and forwarding step by step. And after each agent receives the data, analyzing the content of the experience package and comparing the content with the local environment so as to dynamically adjust local strategy parameters and realize collaborative optimization. According to the vulnerability intelligent collaborative detection method and system based on multi-agent reinforcement learning provided by the invention, the problems that network security vulnerability detection is poor in expandability and lacks knowledge generalization ability are solved.

Description

Technical Field

[0001] This invention relates to the field of computer network security technology, and in particular to a method and system for intelligent collaborative vulnerability detection based on multi-agent reinforcement learning. Background Technology

[0002] Cybersecurity is the cornerstone of ensuring the stable operation of critical information infrastructure, and vulnerability detection is the first and crucial line of defense in a cybersecurity protection system. Traditional vulnerability scanning tools rely on manually maintained knowledge bases for static feature matching, and can only discover known vulnerabilities. With the development of the IoT "cloud-edge-device" architecture and the access of massive heterogeneous terminals, traditional tools face the following two major limitations:

[0003] First, it suffers from poor scalability, i.e., the "curse of dimensionality" problem. Centralized scanning has high scheduling overhead and long cycles across tens of thousands of terminals, making it impossible to perceive the security situation in real time.

[0004] Second, there is a lack of knowledge generalization ability, i.e., the problem of "information silos." Tools operate in isolation and cannot share vulnerability knowledge. For example, a kernel vulnerability discovered on one device cannot be exploited by scanners on other heterogeneous platforms, resulting in "each fighting its own battle," wasting collective intelligence, and passively lagging security protection.

[0005] There is currently no effective solution to the problems of poor scalability and lack of knowledge generalization ability in the detection of network security vulnerabilities in related technologies. Summary of the Invention

[0006] The present invention provides a method and system for intelligent collaborative vulnerability detection based on multi-agent reinforcement learning, which at least solves the problems of poor scalability and lack of knowledge generalization ability in network security vulnerability detection.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] The first aspect of this invention provides a method for intelligent collaborative vulnerability detection based on multi-agent reinforcement learning, comprising the following steps: Each agent deployed at an edge node independently executes vulnerability detection actions based on local observation information and a policy generation model; wherein the policy generation model is constructed based on a decentralized partially observable Markov decision process and obtained through centralized cloud training and collaborative optimization with a globally shared reward function as the optimization objective; when an agent obtains a positive reward through vulnerability detection, the agent encapsulates the vulnerability context and associated policy parameters discovered in this vulnerability detection into an experience package; each agent sends the experience package to a randomly selected group of neighboring nodes, and agents deployed at the neighboring nodes forward the experience package to the remaining neighboring nodes level by level; wherein each neighboring node belongs to the collaborative detection network of the edge node; after receiving the experience package, each agent parses the vulnerability context in the experience package and compares it with its local environment, dynamically adjusting the agent's local policy network parameters to achieve collaborative optimization.

[0009] Preferably, before each agent deployed at the edge node independently executes vulnerability detection actions based on local observation information and a policy generation model, the method further includes the following steps: In the edge IoT system environment, a core tuple is generated based on a decentralized partially observable Markov decision process to construct a pre-trained policy model; wherein, the core tuple includes at least an agent, a global state, an action set, and a global shared reward function; the pre-trained policy model is centrally and collaboratively trained in a cloud training center with the global shared reward function as the optimization objective, so that each agent interacts in the simulation environment to generate joint observation, joint action, and global reward data and uploads them to the cloud to form a global experience dataset; based on the global experience dataset, the global value of the joint actions is evaluated, the contribution ratio of each agent to the global reward is calculated, and the value feedback result is output; based on the value feedback result, the core parameters used by each agent for decision-making detection actions are iteratively adjusted until the joint policy converges, and finally, a policy generation model is obtained.

[0010] Preferably, each agent deployed at the edge node independently executes vulnerability detection actions based on local observation information and a policy generation model, including the following steps: The local observation modules of each agent deployed at the edge node collect local observation information from the edge node and the downstream IoT terminals managed by the edge node, forming a local observation representation; the local observation representation is input into the policy generation model, which outputs a probability distribution within the corresponding vulnerability detection action space; based on the probability distribution, each agent selects the optimal detection action matching the local environment from a preset action set; and each agent executes the optimal detection action to perform vulnerability detection on the local area and the downstream IoT terminals managed by the edge node.

[0011] Preferably, when the agent receives a positive reward for vulnerability detection, the agent encapsulates the vulnerability context and associated policy parameters discovered during the vulnerability detection into an experience package, including the following steps: when the agent discovers a previously unknown vulnerability during the vulnerability detection process, it receives a positive reward; when the agent repeatedly discovers a previously discovered vulnerability during the vulnerability detection process, it receives a negative reward; when the agent receives a positive reward, it automatically extracts the vulnerability context information and associated policy parameters corresponding to the vulnerability; wherein, the vulnerability context information includes at least: host operating system type, central processing unit architecture, and the service that triggered the vulnerability; according to a preset data format, the vulnerability context information, associated policy parameters, and core identifier are integrated to generate structured data; wherein, the core identifier includes at least: a globally unique experience package identifier, a source agent identifier, a timestamp, and a vulnerability type identifier; the structured data is subjected to feature extraction and encoding processing, and then encapsulated to obtain the experience package.

[0012] Preferably, the structured data is subjected to feature extraction and encoding processing, and then packaged to obtain the experience package, comprising the following steps: extracting features from the structured data to generate core features; wherein, the core features include at least: vulnerability triggering condition features, device environment adaptation features, key configuration features of detection strategies, and vulnerability type features; converting the core features into feature vectors of a unified format, and eliminating data format differences in heterogeneous environments through encoding processing; associating and encapsulating the feature vectors with the core identifier and the vulnerability context to generate the experience package.

[0013] Preferably, the experience package is sent to a randomly selected group of neighboring nodes by each of the aforementioned agents, and the experience package is forwarded to the remaining neighboring nodes by agents deployed on the neighboring nodes. This includes the following steps: the agent that generates the experience package randomly selects a preset number of neighboring nodes and sends the experience package to the agents deployed on the neighboring nodes; the agent receiving the experience package verifies and deduplicates the globally unique identifier in the experience package and stores it, excluding the direct sender node, and continues to randomly forward it to other neighboring nodes; the verification, storage, and forwarding process is repeated until no new agent receives the experience package within a preset number of consecutive rounds, thus completing the effective diffusion of the experience package across the entire network.

[0014] Preferably, the agent receiving the experience packet verifies and deduplicates the globally unique identifier in the experience packet and stores it. After excluding the direct sender node, it continues to randomly forward the packet to other neighbor nodes. This includes the following steps: the agent receiving the experience packet extracts the globally unique identifier of the experience packet and queries its own local experience cache; if the globally unique identifier already exists in the local experience cache, the experience packet is discarded directly; if it does not exist, the experience packet is stored in the local experience cache and the receiving status is marked; the agent removes the direct sender neighbor node of the experience packet from its own maintained list of neighbor nodes, randomly selects a preset number of other neighbor nodes, and forwards the experience packet to the agents deployed on the other neighbor nodes.

[0015] Preferably, after receiving the experience package, each agent parses the vulnerability context in the experience package and compares it with its local environment, dynamically adjusting the agent's local policy network parameters to achieve collaborative optimization. This includes the following steps: After receiving the experience package, each agent parses the vulnerability context information in the experience package; each agent compares the parsed vulnerability context information with its corresponding configuration in its local environment to determine the data format differences under heterogeneous environments, and determines the compatibility correlation between the local environment and the vulnerability; if a compatibility correlation is determined, the corresponding agent dynamically adjusts its local policy network parameters based on the associated policy parameters and vulnerability triggering condition characteristics in the experience package; after adjustment, each agent performs mutation testing for the local adaptation scenario, achieving collaborative optimization of detection strategies across edge domains and generalization of vulnerability knowledge.

[0016] Another aspect of this invention provides a vulnerability intelligent collaborative detection system based on multi-agent reinforcement learning, applied to the aforementioned vulnerability intelligent collaborative detection method based on multi-agent reinforcement learning. The system includes: a detection module, configured in each of the agents deployed at edge nodes, which independently executes vulnerability detection actions based on local observation information and a policy generation model to obtain detection information; wherein the policy generation model is constructed based on a decentralized partially observable Markov decision process and obtained through centralized cloud training and collaborative optimization with a globally shared reward function as the optimization objective; and an encapsulation module, configured in each of the agents and connected to the detection module, used to, based on the detection information, when an agent obtains a positive reward through vulnerability detection... During the activation process, the agent encapsulates the vulnerability context and associated policy parameters discovered during the vulnerability detection into an experience package. A transceiver module, located in each agent, sends the experience package to a randomly selected group of neighboring nodes. Agents deployed on these neighboring nodes forward the experience package to the remaining neighboring nodes and receive experience packages from them. Each neighboring node belongs to the collaborative detection network of the edge node. An optimization module, also located in each agent, parses the vulnerability context in the experience packages from other agents and compares it with the local environment, dynamically adjusting the agent's local policy network parameters to achieve collaborative optimization.

[0017] A third aspect of the present invention provides an electronic device, comprising: a processor and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the aforementioned intelligent collaborative vulnerability detection method based on multi-agent reinforcement learning.

[0018] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0019] The present invention provides a method and system for intelligent collaborative vulnerability detection based on multi-agent reinforcement learning. First, it relies on a globally generated policy model trained centrally in the cloud, combined with independent and parallel vulnerability detection by agents at each edge node based on local observation information. This achieves a paradigm of centralized training and distributed execution, fundamentally overcoming the scalability bottleneck and response latency caused by centralized scheduling—the curse of dimensionality. Second, when any agent successfully detects a vulnerability, it encapsulates the specific vulnerability context and associated policy parameters into an experience package, which is then chained across the network through random neighbor selection. Other agents receive this package and dynamically fine-tune their own policies by comparing their local environment with the vulnerability context, achieving a continuous learning loop. This allows agents to migrate and adapt their policies online based on shared experience, extending reinforcement learning from offline training to online collaboration. This ensures that the detection knowledge obtained at a specific node can be quickly absorbed and reused by other nodes with similar vulnerabilities, breaking down information silos between devices and achieving cross-node generalization and proactive immunity of security threat characteristics. This solves the problems of poor scalability and lack of knowledge generalization ability in network security vulnerability detection. Attached Figure Description

[0020] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other embodiments based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating an embodiment of the intelligent collaborative vulnerability detection method based on multi-agent reinforcement learning, as described in this invention.

[0022] Figure 2 This is a schematic diagram of the structure of a vulnerability intelligent collaborative detection system based on multi-agent reinforcement learning, an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram illustrating the core steps of the intelligent collaborative vulnerability detection method based on multi-agent reinforcement learning, as described in an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram of the experience sharing and collaborative optimization part of the vulnerability intelligent collaborative detection method based on multi-agent reinforcement learning in an embodiment of the present invention.

[0025] Figure 5 This is a schematic diagram of the structure of the electronic device created by this invention. Detailed Implementation

[0026] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0027] The standard practice for network vulnerability scanning is to deploy commercial or open-source vulnerability scanning tools. The technical paradigms of vulnerability scanning tools in related technologies are essentially similar: they primarily rely on maintaining a large, manually updated vulnerability knowledge base. The scanner uses this knowledge base to perform periodic, static scans of host systems in the target network based on matching known characteristics (such as software version and service fingerprints) to discover known, archived vulnerabilities.

[0028] However, with the development of IoT technology, network architecture is evolving towards an edge IoT system that integrates cloud, edge, and device collaboration. In edge IoT scenarios, massive numbers of heterogeneous IoT terminals (such as sensors and smart meters) are connected to edge gateways. Large enterprises such as State Grid have deployed tens of thousands, or even hundreds of thousands, of terminal devices. These devices are not only numerous but also highly heterogeneous.

[0029] In such a large-scale edge IoT environment, the limitations of vulnerability scanning tools in related technologies become particularly prominent, mainly in the following two aspects:

[0030] First, there is poor scalability, or the "disaster of dimensions" problem. Based on a periodic, centralized scanning model, when facing tens of thousands of terminals, the scheduling, execution, and report aggregation of scanning tasks generate huge network and computing overhead. This results in long scanning cycles and delayed responses, making it impossible to achieve near real-time awareness of the overall network security situation, let alone deal with sudden zero-day vulnerability attacks.

[0031] Second, there is a lack of knowledge generalization capability, i.e., the problem of "information silos." This is the most serious deficiency in heterogeneous environments. Scanning tools in related technologies operate in isolation, lacking effective collaboration and information sharing mechanisms. For example, when a scanner discovers an unknown kernel privilege escalation vulnerability in a specific domestically produced IT device using some advanced detection method, this valuable "knowledge" is confined to that scanner. It cannot be understood and utilized by scanners on other heterogeneous platforms in the network, even though these heterogeneous platforms may have vulnerabilities of the same origin caused by the same code logic defects. Each intelligent agent remains in an inefficient "lone wolf" mode, leading to a huge waste of "swarm intelligence," making security protection always passive, lagging, and feature-dependent.

[0032] To address the issues of poor scalability and lack of knowledge generalization ability in network security vulnerability detection, embodiments of this invention provide a method and system for intelligent collaborative vulnerability detection based on multi-agent reinforcement learning.

[0033] Among them, such as Figure 1 As shown, the vulnerability intelligent collaborative detection method based on multi-agent reinforcement learning provided by the embodiments of the present invention includes the following steps S1 to S4.

[0034] Step S1: Each agent deployed on the edge node independently executes vulnerability detection actions based on local observation information and the policy generation model. The policy generation model is constructed based on a decentralized partially observable Markov decision process and is obtained through centralized cloud training and collaborative optimization with a globally shared reward function as the optimization objective.

[0035] Step S2: When the agent receives a positive reward through vulnerability detection, the agent encapsulates the vulnerability context and associated policy parameters discovered in this vulnerability detection into an experience package.

[0036] Step S3: Each agent sends the experience package to a randomly selected group of neighboring nodes, and the agents deployed on the neighboring nodes forward the experience package to the remaining neighboring nodes level by level; wherein, each neighboring node belongs to the cooperative detection network of the edge node.

[0037] In step S4, after receiving the experience package, each agent parses the vulnerability context in the experience package and compares it with the local environment, dynamically adjusting the agent's local policy network parameters to achieve collaborative optimization.

[0038] An intelligent agent is an autonomous decision-making program or algorithm entity deployed at various edge nodes of a power distribution network. Intelligent agents can be substations, smart meter concentrators, distributed energy controllers, etc. in a power distribution network.

[0039] Local observation information refers to the data that each agent obtains from its edge node and directly connected local network environment. It may include: the edge node's own state data (such as CPU load, memory usage, process list), local network traffic data (such as abnormal connections, specific protocol messages), limited state information of neighboring nodes (within the range of communication allowed), and local logs and vulnerability scan results.

[0040] Decentralized Partially Observable Markov Decision Process (Dec-POMDP) ​​is a mathematical framework for modeling multi-agent collaborative decision-making. Within this framework, each agent can only access local observation information and cannot grasp the complete global state. Decisions are made independently by each agent based on local information, without the need for a central controller. The vulnerability detection process is modeled as a sequential decision problem, where agents select actions (detection behaviors) based on their current state (observations), receive environmental feedback rewards, and transition to a new state.

[0041] The global shared reward function is a unified evaluation criterion used to measure the overall effectiveness of collaborative actions by all agents. During training, although each agent makes independent decisions, they all share the common optimization goal of maximizing this global shared reward function. This means that the agent's policy updates consider not only the immediate benefits of its own actions but also its contribution to the overall effectiveness of collaborative exploration, thereby driving agents to learn to cooperate rather than act independently, ultimately achieving a collaborative optimization effect of "1+1>2".

[0042] The strategy generation model of this invention is constructed based on a decentralized partially observable Markov decision process and is obtained through centralized training in the cloud using a globally shared reward function. This paradigm of distributed execution and centralized training ensures that each agent can respond quickly and autonomously based on local information in real-world edge environments (high real-time performance, low communication overhead, and strong privacy protection). Furthermore, through a global perspective and collaborative optimization training in the cloud, all agents learn a strategy of tacit cooperation to achieve the globally optimal goal, overcoming the limitations of local observation and significantly improving the overall efficiency and intelligence level of large-scale distributed network vulnerability collaborative detection.

[0043] Step S1 of this invention, by deploying agents at each edge node and enabling them to independently perform detection based on local observation information and a collaboratively trained policy generation model, achieves distributed and parallel execution of vulnerability detection capabilities. This avoids the single-point bottleneck, network bandwidth pressure, and privacy leakage risks associated with traditional centralized scanning. Furthermore, since the policy generation model is optimized with a globally shared reward objective, the independent local decisions of each agent implicitly consider collaborative goals. This lays an adaptive, low-latency distributed decision-making foundation for experience sharing and parameter adjustment in subsequent steps, thereby significantly improving detection efficiency, scalability, and robustness in engineering practice.

[0044] Furthermore, each intelligent agent in the embodiments of the present invention can provide positive numerical feedback when it successfully discovers a real and valid vulnerability in order to obtain a positive reward.

[0045] When an agent receives a positive reward, it indicates that the chosen probe action in the current observed state was correct and efficient, thus guiding the agent to be more inclined to choose such actions in similar situations in the future. The ultimate goal is for all agents to collaboratively minimize network-wide security risks by maximizing the long-term accumulated global rewards.

[0046] After a successful detection and positive reward, the agent immediately initiates a knowledge transformation process: it extracts and associates the key elements that led to the success of this interaction, namely the vulnerability context describing the specific conditions for triggering the vulnerability and the associated policy parameters that guide its successful decision-making, from its transient memory, and encapsulates them together into a standardized and transferable data unit (experience package).

[0047] This process not only completes the extraction and compression from raw, high-dimensional local interactive data into refined knowledge objects, but more importantly, it transforms an isolated success event into a complete knowledge carrier containing a mapping relationship between problem scenarios and solutions. This provides the smallest directly processable and transferable knowledge unit for subsequent experience sharing and strategy generalization in collaborative networks.

[0048] Furthermore, in step S3 of this embodiment, after an agent generates an experience packet, the agent first randomly selects several (not all) of its directly connected neighbor nodes as initial transmission targets. Each neighbor node agent that receives the experience packet immediately assumes the role of a new forwarder after receiving it, and it also randomly selects the next batch of nodes from its own set of neighbors (usually excluding the upstream sender) for forwarding.

[0049] This process continues autonomously and in a relay-like manner in the network in a chain or tree structure, with each node avoiding repeated forwarding through the unique identifier of the experience packet, until the experience packet has completed penetration and coverage in the entire collaborative detection network composed of all edge nodes.

[0050] Step S3 of the present invention, through the core processing flow of random neighbor selection and chain relay, enables each node to spread experience packets to the entire network by communicating with only a very small number of neighbors, thereby reducing the huge communication overhead of traditional network broadcasting or centralized distribution to a constant-level load that is independent of network size.

[0051] Due to the randomness and diversity of propagation paths, a single experience package can automatically bypass faulty nodes or blocked links in the network and complete the penetration through other available paths, which greatly enhances the robustness and reliability of the knowledge distribution network. Furthermore, this mechanism is completely decentralized and does not rely on any master node for scheduling, thus fundamentally ensuring the scalability of the entire system.

[0052] Furthermore, in step S4 of this embodiment of the invention, after the agent receives the experience packet from the network, it first parses out the key vulnerability context information (such as specific service version, abnormal load characteristics, etc.) in the packet, and quickly compares it with the current environment state of its own node to determine whether there is a similar vulnerability locally.

[0053] If a matching feature is found, the AI ​​will use the "policy parameters" (such as the probability distribution of successful actions in the corresponding state) attached to the experience package as a high-value reference. Through lightweight transfer learning or parameter fine-tuning algorithms, it will perform targeted and minor updates to its local policy neural network, thereby quickly absorbing and adapting the successful detection strategies of other nodes to the local environment, realizing the instant optimization of individual strategies and the synergistic utilization of group experience.

[0054] The collaborative process using a lightweight transfer learning method includes: First, the agent uses the policy network parameters (such as the weights of some network layers) in the experience package as an initialization baseline or reference model; then, combined with the "shared vulnerability features" determined after comparison with the local environment, it identifies the specific layers or attention modules in the policy network that are most relevant to these features; finally, on these specific modules, it performs very short-term fine-tuning training using a small amount of local interaction data or synthetic samples, so that the policy can maintain its original general detection capabilities while enhancing the accuracy and efficiency of its response to these specific vulnerability scenarios.

[0055] The collaborative process of parameter fine-tuning algorithms includes: the agent first parses the parameters directly related to successful decisions in the experience package (such as the action value vector of the final decision layer or the activation bias of specific neurons) and calculates the difference between them and the corresponding parameters of the local policy; then, a constrained optimization method (such as gradient descent with regularization) is adopted to minimize the difference, while strictly limiting the magnitude and range of parameter updates to ensure that the stability of the original policy in other scenarios is not compromised while incorporating external knowledge; finally, only the selected key weights are iteratively updated to achieve fast and smooth policy improvement.

[0056] The present invention provides a method for intelligent collaborative vulnerability detection based on multi-agent reinforcement learning. First, it relies on a global policy generation model generated by centralized training in the cloud, and combines each edge node agent to perform independent and parallel vulnerability detection based on local observation information. This realizes a paradigm of centralized training and distributed execution, and fundamentally overcomes the scalability bottleneck and response delay caused by centralized scheduling, i.e., the curse of dimensionality problem.

[0057] Secondly, when any agent successfully detects a vulnerability, it encapsulates the specific vulnerability context and associated policy parameters into an experience package, which is then chained across the network via random neighbor selection. Other agents receive this package and dynamically fine-tune their own policies by comparing their local environment with the vulnerability context, achieving a continuous learning loop. This allows agents to migrate and adapt their policies online based on shared experience, extending Multi-Agent Reinforcement Learning (MARL) from offline training to online collaboration. This ensures that the detection knowledge gained at a specific node can be quickly absorbed and reused by other nodes with similar vulnerabilities, breaking down information silos between devices and achieving cross-node generalization and proactive immunity of security threat characteristics. This solves the problems of poor scalability and lack of knowledge generalization ability in network security vulnerability detection.

[0058] Furthermore, the method provided in the embodiments of the present invention further includes the following steps before step S1: In an edge IoT system environment, core tuples are generated based on a decentralized partially observable Markov decision process to construct a pre-trained policy model; wherein, the core tuples include at least an agent, a global state, an action set, and a global shared reward function; the pre-trained policy model is centrally and collaboratively trained in a cloud training center with the global shared reward function as the optimization objective, so that each agent interacts to generate joint observation, joint action, and global reward data in the simulation environment and uploads them to the cloud to form a global experience dataset; based on the global experience dataset, the global value of the joint actions is evaluated, the contribution ratio of a single agent to the global reward is calculated, and the value feedback result is output; based on the value feedback result, the core parameters of each agent used for decision-making and detection actions are iteratively adjusted until the joint policy converges, and finally the policy generation model is obtained.

[0059] The edge IoT system environment is an overall abstract architecture that encompasses all edge computing nodes (such as smart meters, feeder terminals, distributed energy controllers, etc.) in the power distribution network and the network they constitute.

[0060] The cloud utilizes a global perspective to centrally train and collaboratively optimize models. The trained policy models are then distributed to various edge nodes. Each edge node uses the policy to generate models in its local IoT environment for distributed and autonomous vulnerability detection and decision-making. It also synchronizes policies and shares experiences with the cloud and other nodes via the network, thus forming a highly efficient collaborative defense system with centralized optimization in the cloud and distributed execution at the edge.

[0061] The core tuple is defined based on a decentralized partially observable Markov decision process and includes at least: an agent, a global state, a set of actions, and a globally shared reward function, which can be represented as: .

[0062] in, Represents a node in the state space; The global state space is a high-dimensional tensor that represents the true safe state of N nodes in the network at k time points.

[0063] Represents intelligent agents The action space is defined as the set of actions that an agent can execute. This set is divided into multiple functional subsets, including at least vulnerability scanning actions for obtaining target information, fuzzing actions for triggering potential defects, and cooperative communication actions for exchanging information.

[0064] Let represent the state transition probability function, which describes the probability distribution of the environment transitioning to the next global state given the current global state and the joint actions of all agents.

[0065] Represent the local observation space and define the intelligent agent. The sensor readings.

[0066] The reward function is key to guiding multi-agent collaborative learning, encouraging agents to pursue "global optimum" rather than "individual optimum".

[0067] The observation function specifies the probability distribution of each agent i obtaining its specific local observations given a global state. This models the inherent limitations and partial observability of each edge node sensor (or monitoring tool).

[0068] The discount factor is a scalar parameter between 0 and 1. ∈[0,1), used to weigh the importance of the immediate reward obtained by the agent against the future reward. The closer a value is to 1, the more the strategy optimization considers long-term returns; the closer it is to 0, the more it focuses on short-term gains.

[0069] After generating the core tuple, the observation space and action space defined by the core tuple are used to initialize the corresponding policy neural network for each agent, thereby constructing a pre-trained policy model with parameters to be optimized.

[0070] Specifically, in the cloud training center, a centralized collaborative training of the pre-trained policy model is performed with the global shared reward function as the optimization objective. The process is as follows: The training center first constructs a high-fidelity simulation environment that fully simulates the dynamics of the edge IoT system and maintains a fully perceptible global state. Each agent is loaded with the current version of the policy network parameters and runs in parallel within the simulation. At each time step, the simulation environment generates observations reflecting only local information for each agent based on its internal state and dynamics. Each agent independently selects an action based on these observations, and all actions are aggregated to form a joint action. After executing this joint action, the simulation environment updates the global state according to preset state transition rules and calculates the global reward obtained in that step based on the global shared reward function. Simultaneously, it generates new local observations for the next time step, thus achieving centralized collaborative training of the pre-trained policy model.

[0071] Subsequently, the system collects and stores the complete data generated from this round of interaction, including the joint observations, joint actions, global rewards obtained, and the next joint observation by all agents, as an experience trajectory in the global experience replay pool. This "observation-decision-execution-storage" cycle is carried out on a large scale and continuously in the simulation environment, thereby efficiently accumulating massive amounts of collaborative interaction trajectories and forming a global experience dataset for subsequent value assessment and policy iteration optimization.

[0072] Furthermore, based on the global experience dataset, evaluating the global value of a joint action involves estimating its global value using a centralized commentator network, based on joint observations and joint actions within the global experience dataset.

[0073] Subsequently, to calculate the contribution of a specific agent, a counterfactual baseline method is used: while keeping the actions of all other agents unchanged, the agent's actual action is replaced with a baseline action (such as the average action or the default action), and a comparative value is re-estimated.

[0074] The agent's contribution to the global reward, or its marginal contribution, can be quantified by the difference between the global value of its actual actions and the counterfactual baseline value. This difference, as an individualized advantage signal, effectively distinguishes the specific role of each agent in the collective outcome, thereby guiding its strategy to be updated in a direction that benefits global collaboration.

[0075] The output value feedback result is an individual advantage signal tailored for each agent i. It is usually represented as a scalar value that can quantify the additional increment or marginal contribution to the global joint value brought by a specific action taken by agent i at a specific moment compared to when it takes a normal or average action.

[0076] Furthermore, based on the value feedback results, the core parameters for each agent to make decision-making and detection actions are iteratively adjusted as follows: In each training iteration, after the centralized commentator network calculates the individual advantage value of each agent's action, each agent's policy network is independently updated according to the policy gradient theorem.

[0077] The policy gradient direction is guided by the weighted average of individual advantage values. The gradient ascent algorithm is used to adjust the parameters of the policy network (such as the weights of the neural network). In essence, it increases the probability of choosing actions that bring positive advantage values ​​and decreases the probability of actions that bring negative advantage values. This update is usually achieved by minimizing a loss function that combines the advantage function and the policy probability ratio, and by using stochastic gradient descent and backpropagation to optimize the parameters, so that the policy of each agent evolves in a direction that continuously improves its contribution to the global reward.

[0078] When the policies of all agents collectively reach a stable equilibrium state, it is called joint policy convergence. In this state, any single agent changing its policy can no longer improve the long-term cumulative return of the team as a whole. It is usually judged by the following indicators: 1) The average global reward curve obtained by the agent group in multiple evaluation cycles enters a stable range and no longer increases significantly; 2) The magnitude of the update of the network parameters of each agent's policy (gradient norm) is consistently lower than a set threshold; 3) The entropy value of the agent action selection distribution tends to stabilize, indicating that the randomness of the policy is reduced and stabilized; 4) In an independent test environment set, the policy performance reaches the preset success rate and the fluctuation is minimal.

[0079] The method provided in the embodiments of this invention constructs a simulation training environment with a global perspective, adopts a centralized commentator network and credit allocation mechanism, uses a global experience dataset to evaluate the value of joint actions, and accurately calculates the contribution ratio of a single agent to the global reward, thereby decomposing the abstract global goal into individualized value feedback signals that can guide the independent policy updates of each agent.

[0080] This process enables all agents to collaboratively and iteratively update their policy parameters in a direction that maximizes the global shared reward. This not only overcomes policy bias and short-sighted, individualistic behavior caused by distributed local observations, but also provides crucial support for generating a policy generation model that is highly efficient and internally consistent in real-world edge environments through theoretical convergence guarantees. This is equivalent to pre-implanting a globally optimized "cooperative gene" into the entire distributed detection system, ensuring that when performing distributed execution in real, incompletely observable edge environments, the agents still exhibit highly coordinated collective intelligence.

[0081] Further, step S1 of the present invention preferably includes the following steps: collecting local observation information of the edge node and the downstream IoT terminals managed by the edge node through the local observation modules of each intelligent agent deployed on the edge node, forming a local observation representation; inputting the local observation representation into the policy generation model, and outputting the probability distribution in the corresponding vulnerability detection action space by the policy generation model; based on the probability distribution, selecting the optimal detection action matching the local environment from the preset action set by each intelligent agent; and performing vulnerability detection on the local area and the downstream IoT terminals managed by the edge node by executing the optimal detection action by each intelligent agent.

[0082] Specifically, the local observation module of each edge node agent independently and continuously collects real-time operating data (such as process status, network traffic, and log features) of its node and all downstream IoT terminals, and processes it into a structured local observation representation, namely, key state features extracted within a specific time window.

[0083] Subsequently, the local observation representation is input into a lightweight policy generation model that has been pre-trained and deployed locally. This model outputs a probability distribution in the vulnerability detection action space in an instantaneous manner using forward reasoning, which identifies the effectiveness weight of different actions in historical experience.

[0084] Each agent autonomously selects the optimal detection action from a pre-set action set that best matches the current context and has the highest expected return, based entirely on this probability distribution and in conjunction with real-time analysis of the characteristics of the current local environment (such as active service types).

[0085] Ultimately, the intelligent agent executes this selected action, translating it into specific scanning, testing, or interaction instructions, independently performing precise vulnerability detection on the local node and all IoT terminals under its jurisdiction, without requiring real-time communication with the cloud or any central node.

[0086] The method described in this invention enables a seamless transition from system training to deployment, condensing the complex collaborative knowledge obtained from cloud training into a lightweight policy network distributed across edge nodes. In actual operation, each agent relies solely on local observations for millisecond-level decision-making and execution, completely eliminating reliance on centralized scheduling and computation. This purely distributed, autonomous operating architecture fundamentally avoids communication bottlenecks and single points of failure that may arise from central nodes, resulting in extremely low system decision latency. Furthermore, its overall processing capacity can scale linearly with the number of edge nodes, thus fundamentally solving the "curse of dimensionality" problem in massive terminal scenarios in engineering practice, while ensuring the real-time nature, privacy, and overall robustness of vulnerability detection.

[0087] Further, step S2 of the present invention preferably includes the following steps: when the agent discovers a previously unknown vulnerability during vulnerability detection, it obtains a positive reward; when the agent repeatedly discovers a previously discovered vulnerability during vulnerability detection, it obtains a negative reward; when the agent obtains a positive reward, it automatically extracts the vulnerability context information and associated policy parameters corresponding to the vulnerability; wherein, the vulnerability context information includes at least: host operating system type, central processing unit architecture, and the service that triggered the vulnerability; according to a preset data format, the vulnerability context information, associated policy parameters, and core identifier are integrated to generate structured data; wherein, the core identifier includes at least: a globally unique experience package identifier, a source agent identifier, a timestamp, and a vulnerability type identifier; the structured data is subjected to feature extraction and encoding processing, and then encapsulated to obtain an experience package.

[0088] Specifically, during vulnerability detection, the reward for an agent's actions is determined in real-time by the environment (or simulator) based on a predefined globally shared reward function. The specific rules are as follows: when an agent successfully discovers a previously unknown new vulnerability, it receives a positive reward, signaling a valuable successful exploration; conversely, if its actions merely repeat the discovery of a recorded or known vulnerability, it receives zero or negative rewards, thus suppressing ineffective work and guiding the exploration direction. Only when a significant positive reward is obtained will the subsequent knowledge encapsulation process be automatically triggered, ensuring that the experience package contains high-value, novel discoveries.

[0089] Once a positive reward is triggered, the agent will immediately extract two types of core information from the "memory" of its successful interaction: first, vulnerability context information, which defines the objective characteristics of the environment in which the vulnerability exists, including at least: the type of host operating system that triggered the vulnerability (such as the Linux 5.4 kernel), the central processing unit architecture (such as the ARM v8 architecture), and the specific service / application (such as Apache HTTP Server version 2.4.49); second, associated policy parameters, which are the key states or decision bases of the agent's policy network when making the decision to conduct this successful detection, such as the activation value of the neuron that led to the selection of this action, the advantage function value, or the action probability distribution, which reveals the decision logic of "how to act under what circumstances".

[0090] Subsequently, following a pre-defined standardized data format (such as JavaScript object notation or protocol buffers), the extracted vulnerability context and associated policy parameters are integrated with a set of core metadata identifiers to generate a structured data object. These core identifiers include at least: a globally unique experience package identifier (used for deduplication and tracing within the network), a source agent identifier (identifying the knowledge contributor), a precise timestamp (recording the discovery time), and a vulnerability type identifier (such as a general vulnerability disclosure number or vulnerability category code), thereby ensuring that each experience package has complete traceability and resolvability.

[0091] Finally, feature extraction and encoding are performed on this structured data object. For example, text-based operating system names and service versions are converted into standard numerical codes or embedding vectors, and policy parameters are normalized to ensure data format consistency and efficiency in subsequent processing. After encoding, all information is encapsulated into a compact, serialized binary or text data packet, which is ultimately a data packet that can be transmitted and shared over the network.

[0092] The method described in this invention, through a well-defined structured encapsulation process (integrating vulnerability context, policy parameters, and core metadata), ensures the accurate extraction of key features with strong generalization indicative power (such as operating system and service version) and reusable decision logic (policy parameters) from the original interaction data. This provides high-quality, traceable input for subsequent collaborative learning in the network. This not only achieves efficient knowledge transfer from individuals to groups but also ensures that the receiver can accurately compare local environments and adapt policies based on a unified format. It is a fundamental enabling element that drives the entire system to achieve cross-node, adaptive knowledge generalization.

[0093] Furthermore, feature extraction and encoding are performed on the structured data, and the resulting experience package is encapsulated. This includes the following steps: feature extraction is performed on the structured data to generate core features; the core features include at least: vulnerability triggering condition features, device environment adaptation features, key configuration features of detection strategies, and vulnerability type features; the core features are converted into feature vectors in a unified format, and data format differences in heterogeneous environments are eliminated through encoding; the feature vectors are associated with core identifiers and vulnerability contexts and encapsulated to generate the experience package.

[0094] Specifically, vulnerability triggering condition characteristics are the key combination of conditions extracted from the vulnerability context (such as service version, specific payload) that leads to the successful triggering of the vulnerability.

[0095] Device environment adaptation features encode information such as host operating system type and central processing unit architecture into standardized identifiers that characterize the device platform environment.

[0096] The key configuration features of the detection strategy are extracted from the associated strategy parameters to identify the key configurations or states (such as the activation modes of specific layers) within the strategy network that directly affect the success of this decision.

[0097] Vulnerability type characteristics are vectors of vulnerability categories and severity levels summarized based on vulnerability type identifiers (such as common vulnerability disclosure numbers).

[0098] Through an embedding layer or a standardized encoder, the aforementioned heterogeneous core features are transformed into feature vectors with uniform dimensions and numerical ranges. This encoding process aims to eliminate differences in format and scale between data from different vendors, architectures, or software versions, achieving semantic alignment.

[0099] Finally, the generated unified feature vector is associated with the essential identifiers (such as experience package ID and timestamp) and necessary original vulnerability context from the original data, and then encapsulated into the final data package. This generates an experience package that contains both refined knowledge representation and retains complete traceability information. This package serves as a high-quality, standardized input to drive subsequent cross-node policy migration and collaborative optimization.

[0100] The method described in the embodiments of this invention strips away and encodes complex success experiences that originally depended on specific devices, versions, and instantaneous policy states into a unified feature vector. This process not only eliminates the heterogeneity of different edge nodes in terms of hardware and software environments but also generates a generalizable knowledge pattern that can be directly parsed, compared, and computed by machines. This "generalizable knowledge pattern" enables any node in the network to ignore differences in the underlying environment, quickly determine the correlation between external experience and its own vulnerabilities, and accurately absorb effective policy configurations from it. This fundamentally ensures the lossless flow and efficient collaboration of security knowledge among groups, achieving a qualitative leap from individual experience to herd immunity.

[0101] Furthermore, step S3 of the present invention preferably includes the following steps: the agent generating the experience package randomly selects a preset number of neighboring nodes and sends the experience package to the agents deployed on the neighboring nodes; the agent receiving the experience package verifies and deduplicates the globally unique identifier in the experience package and stores it, and after excluding the direct sender node, continues to randomly forward it to other neighboring nodes; the verification, storage, and forwarding process is repeated until no new agent receives the experience package within a preset number of consecutive rounds, thus completing the effective diffusion of the experience package across the entire network.

[0102] Specifically, the agent that generates the experience package randomly selects a preset number of targets from its neighboring nodes and sends the experience package, which encapsulates the vulnerability context, policy parameters, and core identifiers, to these nodes.

[0103] The agent receiving the experience package first verifies its globally unique identifier to remove duplicates, then parses the vulnerability source context in the experience package (such as LoongArch architecture, KylinOS) and compares it with its own local environment (such as x86_64 architecture, UOS) to identify platform heterogeneity.

[0104] By analyzing the feature vectors and service names in the experience package, the agent identifies suspicious components with similar functions (such as kernel modules with the same name) that exist locally, and dynamically adjusts its local policy network parameters accordingly, which significantly improves the probability of targeted fuzz testing for the local component in subsequent decisions.

[0105] If the adjusted strategy successfully reproduces the vulnerability in the local heterogeneous environment, the agent will receive a positive reward and generate an experience package that records the new environment context (x86_64 / UOS).

[0106] The newly generated experience packets will follow the same random neighbor selection and forwarding mechanism to enter a new round of diffusion, thereby realizing the continuous generalization and propagation of vulnerability knowledge from the source heterogeneous environment to other similar vulnerable nodes across the network.

[0107] The method described in this invention provides a distributed learning mechanism based on experience package parsing, heterogeneous environment comparison, and dynamic strategy adjustment. This system enables the accurate transfer and reuse of security knowledge across heterogeneous edge nodes across architectures and systems. Even with vastly different platforms, agents can still identify functionally similar vulnerable components through feature vector matching and adaptively adjust their local detection strategies, thereby rapidly generalizing the detection capabilities of specific vulnerabilities to previously isolated heterogeneous environments. As multiple agents continuously verify, generate, and disseminate new environment adaptation experiences, local successes continuously spread, evolve, and strengthen within the network, ultimately breaking down "information silos" and fostering the emergence of "swarm intelligence" capable of collaborative perception, autonomous learning, and joint defense across heterogeneous platforms.

[0108] Furthermore, in the embodiments of the present invention, the intelligent agent receiving the experience packet verifies and deduplicates the globally unique identifier in the experience packet and stores it. After excluding the direct sender node, it continues to randomly forward the packet to other neighbor nodes. Preferably, this includes the following steps: the intelligent agent receiving the experience packet extracts the globally unique identifier of the experience packet and queries its own local experience cache; if the globally unique identifier already exists in the local experience cache, the experience packet is directly discarded; if it does not exist, the experience packet is stored in the local experience cache and the receiving status is marked; the direct sender neighbor node of the experience packet is removed from the neighbor node list maintained by itself, and a preset number of other neighbor nodes are randomly selected to forward the experience packet to the intelligent agents deployed on the other neighbor nodes.

[0109] Specifically, the agent receiving the experience packet first extracts the globally unique identifier from the experience packet and performs a quick lookup in its maintained local experience cache (usually based on a hash table). If the identifier already exists, the experience packet is immediately discarded to avoid duplicate processing. If the identifier does not exist, the experience packet is stored completely in the local cache and marked as "received / pending processing". Subsequently, the agent removes the direct source node of this experience packet from its dynamically maintained list of neighbor nodes, and then, based on a preset forwarding count, selects a corresponding number of targets from the remaining neighbor nodes using a random algorithm, and forwards the experience packet to the agents deployed on these target nodes.

[0110] The method described in this invention constructs an efficient and robust distributed knowledge diffusion network through a deduplication mechanism based on globally unique identifiers and a random neighbor forwarding strategy after removing the sender. This prevents experience packets from endlessly circulating and forming a "broadcast storm," significantly reducing unnecessary network communication overhead and processing load. Furthermore, the randomness ensures the diversity and redundancy of knowledge propagation paths, allowing critical vulnerability intelligence to bypass potentially faulty nodes or blocked links, ultimately covering all relevant nodes in the network with a high probability. Thus, under conditions of no centralized scheduling, reliable, low-cost, and self-organized diffusion of security knowledge in a collaborative detection network is achieved, laying a solid communication foundation for collaborative optimization of swarm intelligence.

[0111] Furthermore, step S4 of the present invention preferably includes the following steps: after each agent receives the experience package, it parses the vulnerability context information in the experience package; each agent compares the parsed vulnerability context information with the corresponding configuration of its local environment to determine the data format differences under heterogeneous environments, and determines the compatibility correlation between the local environment and the vulnerability; if a compatibility correlation is determined, the corresponding agent dynamically adjusts the local policy network parameters based on the associated policy parameters and vulnerability triggering condition characteristics in the experience package; after adjustment, each agent performs mutation testing for the local adaptation scenario to achieve cross-edge domain detection strategy collaborative optimization and vulnerability knowledge generalization.

[0112] Specifically, after parsing the experience package, the agent maps the vulnerability context information and local environment configuration into a unified feature space for comparison. By calculating the cosine similarity between feature vectors or the semantic relevance score based on the pre-trained embedding model, the agent quantifies the differences in data format under heterogeneous environments and determines the adaptation relevance.

[0113] If the relevance score exceeds a preset threshold, the AI ​​will employ fine-tuning techniques from transfer learning. Using the association policy parameters in the experience package as prior knowledge, it will dynamically adjust the parameters by updating the gradients of specific layers of the policy network using a small number of local samples. The adjusted AI will then use a fuzzing engine to generate variant test cases tailored to the local context based on the vulnerability triggering conditions, and perform targeted probing.

[0114] The preset threshold is set based on historical data verification and the system's acceptable generalization risk. For example, when the experience package differs from the local environment's CPU architecture (such as ARM versus x86), binary-level vulnerabilities are extremely difficult to migrate, so the system sets a high threshold (e.g., similarity greater than 0.9) to determine relevance, avoiding invalid probes. When the experience package is similar to the core services running in the local environment (such as Apache version number), due to the high probability of software vulnerability reuse, the system sets a lower threshold (e.g., similarity greater than 0.6) to determine relevance, promoting rapid knowledge generalization.

[0115] The steps provided in the embodiments of this invention, by mapping heterogeneous environmental information to a unified feature space for quantitative comparison, enable the system to accurately identify hidden vulnerability associations between different platforms, overcoming misjudgments caused by data format differences in traditional methods. Based on this, policy fine-tuning based on transfer learning allows the agent to quickly absorb successful external experiences and transform them into customized detection capabilities for the local environment. This closed-loop process of "analysis-comparison-adjustment-verification" not only significantly improves the efficiency and accuracy of vulnerability knowledge from individual discovery to group generalization, but also enables the entire system to continuously and automatically evolve cross-platform collaborative defense capabilities in a fully distributed manner, ultimately forming a highly adaptive swarm intelligence security protection system.

[0116] like Figure 2 As shown in the embodiment of the present invention, the intelligent collaborative vulnerability detection system based on multi-agent reinforcement learning is applied to the intelligent collaborative vulnerability detection method based on multi-agent reinforcement learning, and includes: a detection module, an encapsulation module, a transceiver module, and an optimization module.

[0117] The detection module is set up in each intelligent agent deployed on the edge node. Based on local observation information and policy generation model, it independently performs vulnerability detection actions to obtain detection information. The policy generation model is constructed based on a decentralized partially observable Markov decision process and is obtained through centralized cloud training and collaborative optimization with a globally shared reward function as the optimization objective.

[0118] The encapsulation module, located in each agent and connected to the detection module, is used to encapsulate the vulnerability context and associated policy parameters discovered during the vulnerability detection into an experience package when the agent receives a positive reward through vulnerability detection.

[0119] The transceiver module, located in each agent, is used to send experience packets to a randomly selected group of neighboring nodes. The agents deployed on the neighboring nodes forward the experience packets to the remaining neighboring nodes level by level, and receive experience packets from the neighboring nodes. Each neighboring node belongs to the cooperative detection network of the edge nodes.

[0120] The optimization module, set in each agent, is used to parse the vulnerability context in the experience package from other agents and compare it with the local environment, dynamically adjust the agent's local policy network parameters, and achieve collaborative optimization.

[0121] The overall system architecture of this invention follows a centralized training, decentralized execution (CTDE) paradigm. Specifically, this invention includes a centralized training server 100 set up in the cloud, and distributed agent detectors set up on various edge nodes, including a first agent detector 200-1 to an Nth agent detector 200-N.

[0122] The centralized training server 100, located in a cloud-based training center, can physically be a high-performance computing cluster situated in a Security Operations Center (SOC). It plays a core role during the initial "training phase" of system deployment but does not participate in real-time decision-making during the "execution phase." The centralized training server 100 includes: a global state monitor 110, a centralized commentator network 120, and a global experience replay pool 130.

[0123] The global state monitor 110 is responsible for aggregating observation information from all agent detectors in the training environment. ) and action information ( ), used to construct the global state space Approximate to .

[0124] The Centralized Commentator Network 120 is one of the core modules of the CTDE architecture. It is a deep neural network whose input is the joint observations and joint actions of all agents, and whose output is the value assessment of that joint action. Its role is to accurately evaluate the global benefits brought by the joint actions of the agent swarm in a specific state, thereby solving the "reputation allocation" problem in multi-agent environments.

[0125] Global Experience Replay 130 is a large-capacity cache used to store experience tuples reported by all agents. ,in and These are joint observation and joint action. It is a joint observation of the next time step based on the observation function and the new global state S'. It is the instantaneous global reward value calculated based on the global shared reward function, which evaluates the effect of the previous joint action. The value evaluation network 120 is trained using data sampled from this replay pool.

[0126] Distributed intelligent agent detectors, acting as the system's "sensors and limbs," are deployed on each terminal node and are responsible for localized execution. Each detector is a lightweight agent program deployed on various terminal nodes in the network environment. The detector includes a local observation module 210, a decentralized actor network 220, a Gossip communication module 230, and a local experience cache 240.

[0127] The detection module corresponds to the local observation module 210 and the decentralized actor network 220.

[0128] The local observation module 210 is responsible for collecting the original operating status of the node itself and its downstream terminals in real time, and collecting data on the intelligent agent. Local observations It generates a structured local observation representation, including a list of locally open ports, the version number of the running service, CPU / memory / network I / O, and information received from the Gossip communication module 230.

[0129] The decentralized actor network 220 is another core component of the CTDE architecture. It is a lightweight deep neural network whose parameters are set during the training phase by a centralized cloud training center 100. During the execution phase, the network operates solely based on local observations. For input, This represents the local observation of agent i, i∈{1, ..., N}, and the output is in the local action space. ( Let represent the probability distribution over the local action space of agent i, i.e. , This represents the joint action of agent i. This represents the local policy of agent i, which independently makes decisions and executes specific vulnerability detection actions.

[0130] The encapsulation module corresponds to the function of encapsulating experience packets in the Gossip communication module 230, and the transceiver module corresponds to the function of sending, receiving and forwarding experience packets in the Gossip communication module 230.

[0131] The Gossip communication module 230 is responsible for implementing the Gossip protocol, which encapsulates successful detection experiences into experience packets 300, and randomly and efficiently sends and receives experience packets 300 in the peer-to-peer network to achieve decentralized knowledge diffusion.

[0132] The optimization module corresponds to the local experience cache 240, which is used to store important locally discovered experiences and experience packets 300 received from the Gossip network, and to check for duplicates, providing a data foundation for the dynamic adjustment of local strategies.

[0133] The local observation module 210, the decentralized actor network 220, the Gossip communication module 230, and the local experience cache 240 work together to enable the agent to autonomously perceive, make decisions, execute, and learn collaboratively in the local environment.

[0134] like Figure 3As shown, the implementation process of the method in this embodiment of the invention includes: in the initial stage of system deployment, formal modeling of the target network environment is first performed. This invention adopts a decentralized partially observable Markov decision process (Dec-POMDP) ​​because it can accurately describe a dynamic decision-making system composed of multiple agents with incomplete information (i.e., partially observable).

[0135] In the model, the agent Unable to perceive the global true state space It can only obtain local observations through the local observation module 210. Intelligent agents according to Choose one action from the centralized actor network 220 The joint action of all intelligent agents Acting on the environment, causing the environmental state space to change from Transfer to the new state space (State transition function) ), and generate a global reward. At the same time, the environment according to and Generate new joint observations (Observation function) ).

[0136] During the centralized cloud training phase, training takes place in an isolated sandbox environment within a cloud training center. This invention employs an advanced actor-critic algorithm suitable for CTDE, and the specific training process is as follows:

[0137] Step 1: All N agents run in parallel in the simulation environment. Each agent i operates according to its current policy network. Based on private local observation Select and execute an action .

[0138] Step 2, the environment receives this joint action ( The global state S is updated based on its intrinsic dynamics (state transition function P), and then the next joint observation is generated. ), and calculate a global reward value r.

[0139] Step 3: After each group of agents interacts with the environment, a complete experience tuple for all N agents is generated. They are sent and stored in a centralized global experience replay pool.

[0140] Step 4: Randomly sample a batch of experience data from the global experience replay pool. This is based on a centralized commentator network (i.e., a value function). The network learns from this batch of data with the goal of accurately assessing the global value of joint actions given joint observations. The network parameters are updated by minimizing the temporal difference error. Since this critic network possesses global information on all agent observations and actions during training, it can effectively evaluate the value of different joint actions through learning, thus providing a reliable basis for subsequently decomposing the contribution of each agent's action (i.e., solving the credit assignment problem).

[0141] Step 5, each agent Policy network (Parameters are) The policy is updated according to the policy gradient theorem. The gradient direction is guided by a centralized network of critics: aiming to maximize the Q-value of the joint action generated by the current policy, as evaluated by the critics. Through this step, each agent's policy is independently adjusted in a direction that improves the overall long-term return of the team.

[0142] Step 6: Iterate through steps 3 to 5. Training continues until the joint policy formed by the N policy networks stabilizes, meaning that the long-term cumulative discounted reward that the agent group can obtain tends to maximize and no longer significantly increase in multiple rounds of evaluation. At this point, the joint policy can be determined to have converged. Finally, each agent learns to collaboratively achieve the global goal by executing its local policy in a decentralized manner.

[0143] The aforementioned global experience replay pool is the generated global experience dataset, and each data point includes at least: joint observation, joint action, global shared reward, and joint observation at the next time step.

[0144] The implementation process of the Gossip protocol includes the following steps:

[0145] When any agent successfully performs a vulnerability detection action (such as a specific fuzzing test) through its decentralized actor network 220 and receives a positive reward—signifying that it has discovered a new, previously unknown high-risk vulnerability—the system will automatically trigger the following collaborative knowledge dissemination process:

[0146] The Gossip communication module 230 of the first intelligent agent detector 200-1 immediately starts up and encapsulates the key elements of this successful detection, including specific vulnerability context information (such as affected services, operating systems, and triggering conditions), the associated policy parameters that led to the success, and the core metadata identifiers used for tracing, into a structured experience package 300 in a preset format.

[0147] like Figure 4 As shown, after encapsulation, the agent randomly selects a preset number of neighboring agents from its dynamically maintained list of neighboring nodes as initial targets and sends the experience package 300 to them.

[0148] After any agent in the network receives experience packet 300 through its Gossip communication module 230, it first extracts the globally unique identifier (such as packet_id) from the packet and performs a quick lookup in its local experience cache 240. If the identifier already exists, the packet is discarded to prevent circular propagation and handle redundancy; if the identifier does not exist, the packet is stored in the local experience cache 240. Subsequently, the agent removes the sender from its neighbor list and randomly selects q targets from the remaining neighbors to forward experience packet 300 again.

[0149] The aforementioned "receive-verify-forward" process proceeds recursively in a chain-like, exponential manner within the network. Due to the use of the Gossip protocol for randomly selecting neighbors, experience packet 300 propagates through multiple paths within the network, ensuring overall propagation even if some nodes or links fail. This mechanism guarantees that, within a finite timeframe, almost all relevant agents will successfully receive and cache experience packet 300, achieving efficient, reliable, and decentralized sharing of vulnerability knowledge in the collaborative detection network. This lays the data foundation for subsequent adaptive optimization of local policies.

[0150] After caching the experience package, the agent receiving it parses its contents through its local observation module. This module first compares the source vulnerability context recorded in the experience package (e.g., LoongArch architecture, KylinOS operating system) with the agent's own local environment (e.g., x86_64 architecture, UOS operating system), confirming platform heterogeneity. However, by further analyzing the encoded feature vectors and service names in the experience package, the agent identifies a highly similar kernel module running locally, which also exposes similar interfaces (e.g., ioctl).

[0151] Based on this analysis, the agent's policy network uses the parsing results as additional input for policy optimization. The network parameters are dynamically adjusted so that in subsequent decision steps, when the policy network outputs the action probability distribution, it significantly increases the weight of performing fuzzing actions against local specific driver variants (such as driver_variant.ko), especially favoring the reproduction of specific triggering conditions recorded in the experience package. If this targeted test successfully reproduces the vulnerability in the local heterogeneous environment, the agent will receive a positive reward and may package a new experience package containing context information of the current environment (x86_64 / UOS). This new experience package will then initiate a new round of random neighbor propagation.

[0152] Furthermore, the specific implementation algorithm of the actor-critic architecture is not limited to MADDPG; it can also be other MARL algorithms suitable for the CTDE paradigm, such as MAPPO, COMA, and QMIX. The Gossip protocol can also be replaced by other efficient P2P broadcast protocols or DHT (Distributed Hash Table) technology, as long as they can achieve decentralized and eventually consistent experience sharing. The data structure of the experience package can also be added to or removed according to actual security requirements.

[0153] Embodiments of the present invention also provide a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0154] Embodiments of the present invention also provide a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform the method of an embodiment of the present invention.

[0155] An embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method of the embodiment of the present invention.

[0156] In summary, the method and system provided by the embodiments of the present invention have the following beneficial effects:

[0157] 1. Proactivity and Intelligence: This invention introduces Multi-Agent Reinforcement Learning (MARL) to transform vulnerability detection from a passive mode relying on static feature libraries to a proactive predictive mode capable of actively learning and dynamically adjusting scanning strategies. The agents can autonomously learn the optimal detection strategy through interaction with the environment, maximizing the overall vulnerability discovery benefits.

[0158] 2. Scalability (Solving the Curse of Dimension): The "Centralized Training, Decentralized Execution" (CTDE) paradigm adopted in this invention perfectly meets the needs of large-scale networks. Although the training phase is computationally intensive, it is completed in an offline environment; during the execution phase, each agent relies only on local observations and a lightweight policy network to make decisions, without needing to communicate with the central node in real time. This results in extremely high concurrency, low latency, and robustness, and can easily scale to tens of thousands or even hundreds of thousands of terminal nodes.

[0159] 3. Collaboration and Generalization Capabilities (Solving Information Silos): The "experience package" sharing mechanism based on the Gossip protocol designed in this invention is key to breaking down information silos in heterogeneous networks. It allows a local discovery by an agent to be encoded as knowledge containing contextual information (including vectors and policy parameters) and quickly and decentralizedly propagated to other agents. The receiving agent can intelligently "understand" this experience and conduct targeted "mutation tests" in its own heterogeneous environment, thereby achieving rapid knowledge generalization and the emergence of "swarm intelligence," instantly transforming single-point security discoveries into network-wide immunity enhancements.

[0160] 4. Deep Exploration Capability: Through MARL's collaborative exploration and optimized reward function design, this invention can guide intelligent agent clusters to conduct "carpet-bombing" deep mining, and is better at discovering unknown and complex vulnerabilities that are difficult to reach by traditional scanning tools and require multiple steps and complex interactions to trigger.

[0161] refer to Figure 5 The present invention will now describe a structural block diagram of an electronic device that can serve as an embodiment of the present invention, serving as an example of a hardware device applicable to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0162] like Figure 5 As shown, the electronic device includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 502 or loaded from storage unit 508 into RAM (Random Access Memory) 503. RAM 503 can also store various programs and data required for the operation of the electronic device. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An I / O interface (Input / Output Interface) 505 is also connected to bus 504.

[0163] Multiple components in the electronic device are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information into the electronic device. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disks and optical discs. Communication unit 509 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, and / or wireless communication transceivers, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0164] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a CPU (Central Processing Unit), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present invention can be implemented as a computer program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 502 and / or communication unit 509. In some embodiments, the computing unit 501 can be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0165] Computer programs for implementing the methods of embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0166] In the context of embodiments of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0167] It should be noted that the term "comprising" and its variations used in the embodiments of this invention are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "a plurality" mentioned in the embodiments of this invention are illustrative and not restrictive, and those skilled in the art should understand that unless explicitly indicated otherwise in the context, they should be understood as "one or more".

[0168] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0169] The steps described in the method embodiments provided by the present invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0170] The term "embodiment" in this specification refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily imply the same embodiment, nor does it imply independence or alternativeity from other embodiments. The various embodiments in this specification are described in a related manner, with reference to each other for similar or identical parts. In particular, for apparatus, device, and system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, and relevant details are referred to in the description of the method embodiments.

[0171] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A vulnerability intelligent collaborative detection method based on multi-agent reinforcement learning, characterized in that, Includes the following steps: Each agent deployed at the edge node independently performs vulnerability detection actions based on local observation information and a policy generation model. The policy generation model is constructed based on a decentralized partially observable Markov decision process and is obtained through centralized cloud training and collaborative optimization with a globally shared reward function as the optimization objective. When the agent receives a positive reward for vulnerability detection, it encapsulates the vulnerability context and associated policy parameters discovered during the vulnerability detection into an experience package. Each of the aforementioned agents sends the experience package to a randomly selected group of neighboring nodes, and the agents deployed by the neighboring nodes forward the experience package to the remaining neighboring nodes level by level; wherein, each of the aforementioned neighboring nodes belongs to the cooperative detection network to which the edge node is located; After receiving the experience package, each agent parses the vulnerability context in the experience package and compares it with its local environment, dynamically adjusting the agent's local policy network parameters to achieve collaborative optimization.

2. The vulnerability intelligent collaborative detection method based on multi-agent reinforcement learning according to claim 1, characterized in that, Before each agent deployed at the edge node independently executes vulnerability detection actions based on local observation information and policy-generated models, the method further includes the following steps: In the edge IoT system environment, a pre-trained policy model is constructed by generating core tuples based on a decentralized partially observable Markov decision process; wherein, the core tuples include at least an agent, a global state, a set of actions, and a globally shared reward function. In the cloud training center, the pre-trained policy model is centrally and collaboratively trained with the global shared reward function as the optimization objective. This enables each agent to interact and generate joint observation, joint action, and global reward data in the simulation environment and upload them to the cloud to form a global experience dataset. Based on the global experience dataset, evaluate the global value of the joint action, calculate the contribution ratio of each agent to the global reward, and output the value feedback result. Based on the value feedback results, the core parameters of each agent used for decision-making and detection actions are iteratively adjusted until the joint policy converges, and finally the policy generation model is obtained.

3. The intelligent collaborative vulnerability detection method based on multi-agent reinforcement learning according to claim 1, characterized in that, Each agent deployed at the edge node independently performs vulnerability detection actions based on local observation information and policy-generated models, including the following steps: By using the local observation modules of each of the intelligent agents deployed at the edge nodes, local observation information of the edge nodes and the downstream IoT terminals managed by the edge nodes is collected to form a local observation representation; The local observation representation is input into the policy generation model, and the policy generation model outputs the probability distribution in the corresponding vulnerability detection action space. Based on the probability distribution, each of the intelligent agents selects the optimal detection action that matches the local environment from a preset action set. By having each of the aforementioned intelligent agents execute the optimal detection action, vulnerability detection is performed on local and downstream IoT terminals managed by the edge nodes.

4. The intelligent collaborative vulnerability detection method based on multi-agent reinforcement learning according to claim 1, characterized in that, When the agent receives a positive reward for vulnerability detection, it encapsulates the vulnerability context and associated policy parameters discovered during the vulnerability detection into an experience package, including the following steps: When the agent discovers a previously unknown vulnerability during vulnerability detection, it receives a positive reward; when the agent repeatedly discovers a previously discovered vulnerability during vulnerability detection, it receives a negative reward. When the agent obtains a positive reward, it automatically extracts the vulnerability context information and associated policy parameters corresponding to the vulnerability; wherein, the vulnerability context information includes at least: host operating system type, central processing unit architecture, and the service that triggered the vulnerability; According to the preset data format, the vulnerability context information, associated policy parameters and core identifiers are integrated to generate structured data; wherein, the core identifiers include at least: a globally unique experience package identifier, a source agent identifier, a timestamp and a vulnerability type identifier; The structured data is subjected to feature extraction and encoding, and then encapsulated to obtain the experience package.

5. The intelligent collaborative vulnerability detection method based on multi-agent reinforcement learning according to claim 4, characterized in that, The process of extracting and encoding features from the structured data and encapsulating it to obtain the experience package includes the following steps: Feature extraction is performed on the structured data to generate core features; wherein, the core features include at least: vulnerability triggering condition features, device environment adaptation features, key configuration features of detection strategies, and vulnerability type features; The core features are transformed into feature vectors in a unified format, and data format differences in heterogeneous environments are eliminated through encoding processing. The feature vector is associated with and encapsulated with the core identifier and the vulnerability context to generate the experience package.

6. The intelligent collaborative vulnerability detection method based on multi-agent reinforcement learning according to claim 1, characterized in that, The process involves each agent sending the experience package to a randomly selected group of neighboring nodes, and the agents deployed on those neighboring nodes forwarding the experience package to the remaining neighboring nodes in a step-by-step manner, including the following steps: The agent that generates the experience package randomly selects a preset number of neighboring nodes and sends the experience package to the agents deployed on the neighboring nodes. The agent receiving the experience packet verifies and deduplicates the globally unique identifier in the experience packet and stores it. After excluding the direct sender node, it continues to randomly forward it to other neighbor nodes. The process of repeated verification, storage, and forwarding continues until no new intelligent agent receives the experience package within a preset number of consecutive rounds, thus completing the effective dissemination of the experience package across the entire network.

7. The intelligent collaborative vulnerability detection method based on multi-agent reinforcement learning according to claim 6, characterized in that, The agent receiving the experience packet verifies and deduplicates the globally unique identifier in the experience packet and stores it. After excluding the direct sender node, it continues to randomly forward the packet to other neighbor nodes, including the following steps: The agent receiving the experience package extracts the globally unique identifier of the experience package and queries its own local experience cache; If the globally unique identifier already exists in the local experience cache, the experience package is discarded directly; if it does not exist, the experience package is stored in the local experience cache and the receiving status is marked. Remove the direct sender neighbor node of the experience package from the list of neighbor nodes maintained by itself, randomly select a preset number of other neighbor nodes, and forward the experience package to the intelligent agent deployed by the other neighbor nodes.

8. The intelligent collaborative vulnerability detection method based on multi-agent reinforcement learning according to claim 1, characterized in that, After receiving the experience package, each agent parses the vulnerability context in the experience package and compares it with its local environment, dynamically adjusting the agent's local policy network parameters to achieve collaborative optimization, including the following steps: After receiving the experience package, each of the intelligent agents parses the vulnerability context information in the experience package; Each of the aforementioned intelligent agents will compare the parsed vulnerability context information with the corresponding configuration of its own local environment to determine the data format differences in the heterogeneous environment and the compatibility correlation between the local environment and the vulnerability. If an adaptation correlation is determined, the corresponding intelligent agent dynamically adjusts the local policy network parameters based on the associated policy parameters and vulnerability triggering condition characteristics in the experience package. The adjusted agents perform mutation tests for local adaptation scenarios, achieving collaborative optimization of detection strategies and generalization of vulnerability knowledge across edge domains.

9. A vulnerability intelligent cooperative detection system based on multi-agent reinforcement learning, applied to any one of the vulnerability intelligent cooperative detection methods based on multi-agent reinforcement learning as described in claims 1 to 8, characterized in that: include: The detection module is set in each of the intelligent agents deployed on the edge nodes. Based on local observation information and policy generation model, it independently performs vulnerability detection actions to obtain detection information. The policy generation model is constructed based on a decentralized partially observable Markov decision process and is obtained through centralized cloud training and collaborative optimization with a globally shared reward function as the optimization objective. An encapsulation module, set in each of the intelligent agents and connected to the detection module, is used to encapsulate the vulnerability context and associated policy parameters discovered in this vulnerability detection into an experience package based on the detection information when the intelligent agent obtains a positive reward through vulnerability detection. A transceiver module, located in each of the aforementioned agents, is used to send the experience package to a randomly selected subset of neighboring nodes. The agents deployed on the neighboring nodes then forward the experience package to the remaining neighboring nodes level by level, and receive the experience package from the neighboring nodes. Each of the neighboring nodes belongs to the cooperative detection network where the edge node is located. An optimization module, set in each of the intelligent agents, is used to parse the vulnerability context in the experience package from the other intelligent agents and compare it with the local environment, dynamically adjust the local policy network parameters of the intelligent agent, and achieve collaborative optimization.

10. An electronic device, comprising: A processor and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the intelligent collaborative vulnerability detection method based on multi-agent reinforcement learning according to any one of claims 1 to 8.

Images