A Reinforcement Learning-Based Intelligent Security Device Linkage Control Method

Abstract

This invention discloses a linkage control method for intelligent security devices based on reinforcement learning, comprising the following steps: collecting security data in the target security area and splitting the security event trajectory set; performing feature encoding on the security event trajectory set to form a trajectory encoding set; comprehensively evaluating the trajectory encoding set to obtain event trajectories with weighted terminal rewards; inputting the event trajectories with weighted terminal rewards into a reward decomposition model based on the RUDDER algorithm to output a contribution sequence; weighting the terminal rewards according to the contribution sequence to generate a redistribution reward sequence and forming a corresponding linkage control strategy; deploying the linkage control strategy in the security scenario to form online event trajectories; and performing reward decomposition, reward redistribution, and parameter updates based on the online security event trajectories to achieve closed-loop adaptive optimization of the linkage control strategy. This invention realizes linkage control of intelligent security devices.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent security, and particularly to an intelligent security prevention device linkage control method based on reinforcement learning. Background Art

[0002] At present, security devices such as video surveillance, access control, and intrusion alarms have been widely deployed in enterprises, institutions, parks, and urban governance scenarios. To improve the handling efficiency, in engineering, different security devices are usually simply combined by presetting linkage rules. For example, when an intrusion alarm is detected, the camera is automatically linked to turn, the lights are lit, and the access control is locked. Existing systems mostly rely on fixed thresholds, event condition tables, or finite state machines for logical judgment. The linkage logic mainly relies on manual experience configuration, lacking the systematic modeling ability for complex scenarios, temporal information, and the collaborative effects of multiple devices, and it is difficult to adapt to the dynamically changing security environment.

[0003] With the development of machine learning, there are also solutions that attempt to introduce reinforcement learning to optimize some security control strategies. However, existing methods often simplify reinforcement learning into local decision-making learning based on a single device or a single time step. The reward signal design is rough, and indicators such as immediate alarm results and single-shot recognition accuracy are mostly used as feedback, without fully considering the temporal evolution process of the entire security event and the comprehensive impact of multi-device linkage behavior on the final handling result. In addition, when the handling effect can only be evaluated at the end of the event, existing reinforcement learning methods generally have problems such as severely lagging rewards and difficulty in clearly distinguishing the contribution degrees of different device actions at different times, resulting in unstable training and difficult to guarantee the quality of the strategy.

[0004] Therefore, how to provide an intelligent security prevention device linkage control method based on reinforcement learning is an urgent problem for those skilled in the art. Summary of the Invention

[0005] An object of the present invention is to propose an intelligent security prevention device linkage control method based on reinforcement learning. The present invention uses a reinforcement learning framework based on the RUDDER algorithm to decompose and reallocate the rewards for the security event trajectory, finely map the terminal reward to time steps and device actions, train to form a linkage control strategy and continuously update it online, so as to achieve the adaptive linkage control and long-term optimization of intelligent security prevention devices.

[0006] An intelligent security prevention device linkage control method based on reinforcement learning according to an embodiment of the present invention includes the following steps: Collect security data in the target security area, and split the continuous monitoring into a set of security event trajectories arranged in chronological order according to a preset event trigger rule; Feature encoding is performed on the data at each time step in the security event trajectory set, and the linkage behavior of each device is recorded to form a trajectory encoding set including state sequence and action sequence; A comprehensive evaluation of the entire trajectory in the trajectory coding set is performed, and the terminal reward is obtained by combining the scenario risk weighting, resulting in an event trajectory with weighted terminal reward; The event trajectory with weighted terminal feedback is input into the feedback decomposition model based on the RUDDER algorithm. The state sequence and action sequence are sequentially modeled, and the contribution sequence representing the importance of time step and device action is output. The terminal rewards are weighted and allocated according to the contribution sequence to generate a redistribution reward sequence refined to time steps and device actions. At the same time, the linkage decision parameters in the reward decomposition model are trained by combining the state sequence, action sequence and redistribution reward, and a corresponding linkage control strategy is formed. In a security scenario, the aforementioned linkage control strategy is deployed, taking the real-time status as input, outputting linkage control actions of each device and driving their execution, thus forming an online event trajectory. Based on the decomposition of online security event trajectory execution rewards, the redistribution of rewards, and parameter updates, a closed-loop adaptive optimization of the linkage control strategy is achieved.

[0007] Optionally, obtaining the set of security event trajectories specifically includes: Select security devices that participate in the linkage control within the target security area, including video surveillance, access control, and alarms. Set device number, physical location, working mode, and data reporting cycle for each device. Establish data acquisition channels and time stamping rules between the device and the security platform so that each device can stably output raw security data according to a unified time base when the system is running. The data acquisition channel continuously receives raw security data from each security device at a preset cycle. Data reported by different devices at the same time are aligned and merged according to time stamps and stored sequentially in chronological order to form a multi-source security data time series covering the monitoring period. Data from each time step in the multi-source security data time series is read sequentially. Preset event triggering rules are called to determine whether the current time step constitutes a security event. Time steps that are determined to be events are marked with event tags. Time intervals that are continuously marked as events are identified as a security event occurrence process in chronological order. Multi-source security data segments are extracted and merged into a security event trajectory. All trajectories are further sorted according to the start time of each security event trajectory to obtain a set of security event trajectories arranged in chronological order.

[0008] Optionally, the composition of the trajectory encoding set specifically includes: For a set of security event trajectories arranged in chronological order, security event trajectories are selected one by one. Within each trajectory, multi-source security data corresponding to each time step is read sequentially according to time sequence. Security data from different devices within the same time step are classified and merged according to device identifiers and time stamps, and organized into a security data time series with time step as index and original data from multiple devices as content. For each time step, the security data time series obtained by aggregation is cleaned, normalized and format converted. The processed multi-source data is then used to extract features and generate a state feature vector to represent the security scenario at that time step. The vectors are then arranged in chronological order to form a state sequence. While generating the state sequence, the control instructions of each security device are read from the linkage control record using the same time step as the state sequence as the index. The linkage control record is a time-seriesd log of the control instructions and execution status of each security device at each time step. The log is then organized and combined to form an action vector that represents the linkage control behavior at that time step. The action vectors are arranged in chronological order to form an action sequence. The state sequence and action sequence corresponding to each security event trajectory are associated and stored to form a trajectory encoding set including the state sequence and the action sequence.

[0009] Optionally, the process of obtaining the event trajectory with weighted terminal reports specifically includes: For the trajectory code set, security event trajectories are read one by one, and trajectory-level evaluation is performed on the entire trajectory. According to the preset evaluation rules, the state change characteristics, equipment action execution and event end status during the event are quantified. The quantification results are merged according to the parameter configuration of the evaluation rules to obtain the basic evaluation value that reflects the overall handling performance of the event. After obtaining the basic evaluation value, the risk level parameters corresponding to the monitoring area and event category of the current event are extracted from the security system configuration. The risk level parameters are used as weighting factors to perform weighted processing on the basic evaluation value, generating a weighted terminal report for the entire security event trajectory. The weighted terminal report is used to represent the overall quality of the event after comprehensively considering the handling effect, process performance and risk level. The weighted terminal reports are associated with the state sequence and action sequence one by one, and all records are collected and sorted according to the time of the event to form a set of event trajectories with weighted terminal reports.

[0010] Optionally, the output of the contribution sequence representing the importance of time steps and equipment actions specifically includes: Based on the set of event trajectories with weighted terminal rewards, the state sequence, action sequence, and weighted terminal reward uniquely corresponding to the entire trajectory are taken as a set of inputs. A reward decomposition model based on the RUDDER algorithm is introduced. A forward sequence modeling unit that receives state vectors and action vectors in chronological order and outputs forward hidden state sequences is set up, and a reverse reward propagation unit that propagates weighted terminal reward information backward along time from the end of the trajectory is set up. The forward sequence modeling unit and the reverse reward propagation unit interact with each other through a gating structure at each time step, so that each time step retains the accumulated state and action information before that time step and the reward information injected backward from the terminal reward. Within the reward decomposition model, the RUDDER algorithm starts from the initial time step of the security event trajectory. The forward sequence modeling unit receives the state vector and action vector of each time step in chronological order and updates the corresponding forward hidden state at each time step. Meanwhile, the reverse reward propagation unit receives the weighted terminal reward starting from the end of the trajectory and propagates it backward along the time axis. The weighted terminal reward is combined with the forward hidden state and reverse reward information of each time step at the time step level. At each time step, the reward prediction subunit provides the predicted value of the terminal reward of the entire trajectory at the time step truncation. The reward prediction results of adjacent time steps are differentially incremented to obtain the time step contribution signal used to characterize the degree of influence of the time step on the terminal reward. At the same time, the contribution signals of all time steps are normalized and constrained at the entire trajectory level to make the numerical values ​​of the contribution of all time steps consistent with the weighted terminal reward of the trajectory. The time step contribution signal, along with the corresponding time step's state vector, action vector, and the forward hidden state of the current time step, is input into the device action contribution refinement structure set in the model. By matching the embedded representations of different security devices with the time step hidden states, the device-level weights of each security device are calculated. The time step contribution is divided into different device-level contributions according to the device-level weights. Within each device, based on the feature components of the specific linkage control action and the degree of matching between the feature components and the hidden state, the device-level contribution is further decomposed into each specific control action, resulting in a contribution sequence that represents the dual dimensions of the importance of the time step and the device action. This sequence is then associated and stored with the corresponding state sequence, action sequence, and weighted terminal report.

[0011] Optionally, the formation of the linkage control strategy specifically includes: Read the importance contribution sequence covering each time step and the actions of each security device, and sum up all the contribution values ​​belonging to the same security event trajectory to obtain the total contribution of the security event trajectory. This total contribution, together with the weighted terminal reward value and contribution sequence of the current trajectory, serves as the basic data set for executing the weighted allocation of rewards. It is stored according to the trajectory number to ensure the consistency of the trajectory-by-trajectory processing order in the subsequent reward redistribution generation process. In the aforementioned basic data set, for each security event trajectory, each time step in the trajectory contribution sequence and the corresponding actions of each security device are sequentially traversed. The weighted terminal rewards are weighted and split according to the proportion of the contribution in the total contribution of the trajectory. The reward share obtained from the split is used as the redistribution reward value of the current device action. The redistribution reward values ​​of all time steps and all device actions in the same trajectory are arranged in chronological order and device action order to form a redistribution reward sequence refined to time steps and device actions. Together with the corresponding state sequence and action sequence, a trajectory-level sample set for training is constructed. In the reward decomposition model based on the RUDDER algorithm, a linkage decision output unit is set up that shares a hidden representation with the sequence modeling unit and has an independent parameter set. The state sequence of the trajectory-level sample set is input into the sequence modeling unit in chronological order to generate a hidden representation for each time step. The hidden representations of each time step are sent to the linkage decision output unit to obtain the predicted output of the linkage control action of each security device. The predicted output is matched with the redistribution reward of the corresponding time step and the corresponding device action to construct the loss amount for training the parameters of the linkage decision output unit. After each training iteration, the linkage decision parameters are updated according to the change of the loss amount. The linkage decision parameters are the hidden representation. When the preset maximum number of iterations is reached, the training is terminated, and the corresponding linkage control strategy is formed based on the linkage decision parameters obtained at the time of termination.

[0012] Optionally, the formation of the online event trajectory specifically includes: The linkage control strategy is deployed to the security linkage control execution system. A status acquisition channel for receiving real-time data from each security device is configured, and a strategy invocation interface for sending the current real-time status to the linkage control strategy is established. This allows the linkage control strategy to be periodically invoked during the operation of the security scenario. The security linkage control execution system is a comprehensive control platform used to receive real-time status, invoke the linkage control strategy, and send the linkage control actions output by the strategy to each security device to drive its execution. The real-time status consists of multi-source environmental and device operation data collected from each security device at each time step and processed by time synchronization. When running in a security scenario, the current real-time status is obtained from the status acquisition channel according to the preset time step. The linkage control strategy is input through the strategy call interface. The linkage control strategy generates the linkage control actions of each security device in the current time step, and sends them to the corresponding security devices and drives their execution to complete the linkage control in the current time step. While executing the linkage control actions at each time step, the current real-time status, the actual linkage control actions of each security device issued, and the device feedback information are recorded. The data recorded in the continuous time steps are merged according to the trigger time and end time of the security event to form the corresponding online event trajectory.

[0013] Optionally, the implementation of the closed-loop adaptive optimization of the linkage control strategy specifically includes: The online security event trajectories are read one by one, input into the reward decomposition model based on the RUDDER algorithm, and the reward decomposition and reward redistribution are performed to form an online trajectory sample set for updating training. Using the online trajectory sample set as input, the hidden representation generated by the sequence modeling unit is used to iteratively update the internal linkage decision parameters until the preset maximum number of iterations is reached. Based on the updated linkage decision parameters, a new linkage control strategy is formed, realizing closed-loop adaptive optimization of the linkage control strategy.

[0014] The beneficial effects of this invention are: This invention first introduces a reward decomposition model based on the RUDDER algorithm, which decomposes the weighted terminal reward, which can only be obtained at the end of a security event, into a contribution sequence representing the importance of time steps and device actions. This enables a detailed characterization of the complete temporal process of a security event and the role of multi-device linkage behavior, overcoming the shortcomings of existing reinforcement learning schemes, such as severe reward lag and inability to distinguish between critical moments and critical actions.

[0015] This invention weights and distributes terminal rewards based on contribution sequences, generating a redistribution reward sequence refined to time steps and device actions. Within the same model framework, it uses state sequences, action sequences, and redistribution rewards to jointly train linkage decision parameters, forming a linkage control strategy. This allows reward decomposition and strategy learning to proceed collaboratively under a unified structure, improving the consistency between the linkage control strategy and the actual handling results in security scenarios.

[0016] This invention deploys a linkage control strategy in security scenarios and continuously collects online security event trajectories. The online data is then fed back into the reward decomposition and redistribution training closed loop to iteratively update the linkage decision parameters. This enables adaptive optimization of the linkage control strategy when scenarios, risks, and equipment operating states change, thereby achieving a more stable, refined, and continuously optimized control effect in multi-device collaborative linkage and complex time-series security event handling. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a linkage control method for intelligent security devices based on reinforcement learning proposed in this invention; Figure 2 This is a schematic diagram of the algorithm structure of a reinforcement learning-based intelligent security device linkage control method proposed in this invention. Figure 3 This is an algorithm data flow diagram of a reinforcement learning-based intelligent security device linkage control method proposed in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figure 1-3 A method for coordinated control of intelligent security devices based on reinforcement learning includes the following steps: Collect security data in the target security area, and break down continuous monitoring into a set of security event trajectories arranged in chronological order according to preset event triggering rules; Feature encoding is performed on the data at each time step in the security event trajectory set, and the linkage behavior of each device is recorded to form a trajectory encoding set including state sequence and action sequence; A comprehensive evaluation of the entire trajectory in the trajectory coding set is performed, and the terminal reward is obtained by combining the scenario risk weighting, resulting in an event trajectory with weighted terminal reward; The event trajectory with weighted terminal feedback is input into the feedback decomposition model based on the RUDDER algorithm. The state sequence and action sequence are sequentially modeled, and the contribution sequence representing the importance of time step and device action is output. The terminal rewards are weighted and allocated according to the contribution sequence to generate a redistribution reward sequence refined to time steps and device actions. At the same time, the linkage decision parameters in the reward decomposition model are trained by combining the state sequence, action sequence and redistribution reward, and a corresponding linkage control strategy is formed. In a security scenario, the aforementioned linkage control strategy is deployed, taking the real-time status as input, outputting linkage control actions of each device and driving their execution, thus forming an online event trajectory. Based on the decomposition of online security event trajectory execution rewards, the redistribution of rewards, and parameter updates, a closed-loop adaptive optimization of the linkage control strategy is achieved.

[0020] In this embodiment, obtaining the security event trajectory set specifically includes: Select security devices that participate in the linkage control within the target security area, including video surveillance, access control, and alarms. Set device number, physical location, working mode, and data reporting cycle for each device. Establish data acquisition channels and time stamping rules between the device and the security platform so that each device can stably output raw security data according to a unified time base when the system is running. The data acquisition channel continuously receives raw security data from each security device at a preset cycle. Data reported by different devices at the same time are aligned and merged according to time stamps and stored sequentially in chronological order to form a multi-source security data time series covering the monitoring period. Data from each time step in the multi-source security data time series is read sequentially. Preset event triggering rules are called to determine whether the current time step constitutes a security event. Time steps that are determined to be events are marked with event tags. Time intervals that are continuously marked as events are identified as a security event occurrence process in chronological order. Multi-source security data segments are extracted and merged into a security event trajectory. All trajectories are further sorted according to the start time of each security event trajectory to obtain a set of security event trajectories arranged in chronological order.

[0021] In this embodiment, the trajectory encoding set is specifically composed of: For a set of security event trajectories arranged in chronological order, security event trajectories are selected one by one. Within each trajectory, multi-source security data corresponding to each time step is read sequentially according to time sequence. Security data from different devices within the same time step are classified and merged according to device identifiers and time stamps, and organized into a security data time series with time step as index and original data from multiple devices as content. For each time step, the security data time series obtained by aggregation is cleaned, normalized and format converted. The processed multi-source data is then used to extract features and generate a state feature vector to represent the security scenario at that time step. The vectors are then arranged in chronological order to form a state sequence. While generating the state sequence, the control instructions of each security device are read from the linkage control record using the same time step as the state sequence as the index. The linkage control record is a time-seriesd log of the control instructions and execution status of each security device at each time step. The log is then organized and combined to form an action vector that represents the linkage control behavior at that time step. The action vectors are arranged in chronological order to form an action sequence. The state sequence and action sequence corresponding to each security event trajectory are associated and stored to form a trajectory encoding set including the state sequence and the action sequence.

[0022] In this embodiment, the process of obtaining the weighted terminal report event trajectory specifically includes: For the trajectory code set, security event trajectories are read one by one, and trajectory-level evaluation is performed on the entire trajectory. According to the preset evaluation rules, the state change characteristics, equipment action execution and event end status during the event are quantified. The quantification results are merged according to the parameter configuration of the evaluation rules to obtain the basic evaluation value that reflects the overall handling performance of the event. After obtaining the basic evaluation value, the risk level parameters corresponding to the monitoring area and event category of the current event are extracted from the security system configuration. The risk level parameters are used as weighting factors to perform weighted processing on the basic evaluation value, generating a weighted terminal report for the entire security event trajectory. The weighted terminal report is used to represent the overall quality of the event after comprehensively considering the handling effect, process performance and risk level. The weighted terminal reports are associated with the state sequence and action sequence one by one, and all records are collected and sorted according to the time of the event to form a set of event trajectories with weighted terminal reports.

[0023] In this embodiment, the output of the contribution sequence representing the importance of time steps and equipment actions specifically includes: Based on the set of event trajectories with weighted terminal rewards, the state sequence, action sequence, and weighted terminal reward uniquely corresponding to the entire trajectory are taken as a set of inputs. A reward decomposition model based on the RUDDER algorithm is introduced. A forward sequence modeling unit that receives state vectors and action vectors in chronological order and outputs forward hidden state sequences is set up, and a reverse reward propagation unit that propagates weighted terminal reward information backward along time from the end of the trajectory is set up. The forward sequence modeling unit and the reverse reward propagation unit interact with each other through a gating structure at each time step, so that each time step retains the accumulated state and action information before that time step and the reward information injected backward from the terminal reward. Within the reward decomposition model, the RUDDER algorithm starts from the initial time step of the security event trajectory. The forward sequence modeling unit receives the state vector and action vector of each time step in chronological order and updates the corresponding forward hidden state at each time step. Meanwhile, the reverse reward propagation unit receives the weighted terminal reward starting from the end of the trajectory and propagates it backward along the time axis. The weighted terminal reward is combined with the forward hidden state and reverse reward information of each time step at the time step level. At each time step, the reward prediction subunit provides the predicted value of the terminal reward of the entire trajectory at the time step truncation. The reward prediction results of adjacent time steps are differentially incremented to obtain the time step contribution signal used to characterize the degree of influence of the time step on the terminal reward. At the same time, the contribution signals of all time steps are normalized and constrained at the entire trajectory level to make the numerical values ​​of the contribution of all time steps consistent with the weighted terminal reward of the trajectory. The time step contribution signal, along with the corresponding time step's state vector, action vector, and the forward hidden state of the current time step, is input into the device action contribution refinement structure set in the model. By matching the embedded representations of different security devices with the time step hidden states, the device-level weights of each security device are calculated. The time step contribution is divided into different device-level contributions according to the device-level weights. Within each device, based on the feature components of the specific linkage control action and the degree of matching between the feature components and the hidden state, the device-level contribution is further decomposed into each specific control action, resulting in a contribution sequence that represents the dual dimensions of the importance of the time step and the device action. This sequence is then associated and stored with the corresponding state sequence, action sequence, and weighted terminal report.

[0024] In this embodiment, the formation of the linkage control strategy specifically includes: This invention, through forward modeling and backward reward propagation based on RUDDER, finely decomposes the weighted terminal reward obtained only at the end of the trajectory into specific time steps and equipment actions, constructs a dual-dimensional contribution sequence aligned with the depth of state and action, and provides accurate and stable reinforcement learning signals for subsequent reward redistribution and linkage control strategy learning, thereby highlighting the key equipment actions at critical moments and improving the overall handling quality.

[0025] Read the importance contribution sequence covering each time step and the actions of each security device, and sum up all the contribution values ​​belonging to the same security event trajectory to obtain the total contribution of the security event trajectory. This total contribution, together with the weighted terminal reward value and contribution sequence of the current trajectory, serves as the basic data set for executing the weighted allocation of rewards. It is stored according to the trajectory number to ensure the consistency of the trajectory-by-trajectory processing order in the subsequent reward redistribution generation process. In the aforementioned basic data set, for each security event trajectory, each time step in the trajectory contribution sequence and the corresponding actions of each security device are sequentially traversed. The weighted terminal rewards are weighted and split according to the proportion of the contribution in the total contribution of the trajectory. The reward share obtained from the split is used as the redistribution reward value of the current device action. The redistribution reward values ​​of all time steps and all device actions in the same trajectory are arranged in chronological order and device action order to form a redistribution reward sequence refined to time steps and device actions. Together with the corresponding state sequence and action sequence, a trajectory-level sample set for training is constructed. In the reward decomposition model based on the RUDDER algorithm, a linkage decision output unit is set up that shares a hidden representation with the sequence modeling unit and has an independent parameter set. The state sequence of the trajectory-level sample set is input into the sequence modeling unit in chronological order to generate a hidden representation for each time step. The hidden representations of each time step are sent to the linkage decision output unit to obtain the predicted output of the linkage control action of each security device. The predicted output is matched with the redistribution reward of the corresponding time step and the corresponding device action to construct the loss amount for training the parameters of the linkage decision output unit. After each training iteration, the linkage decision parameters are updated according to the change of the loss amount. The linkage decision parameters are the hidden representation. When the preset maximum number of iterations is reached, the training is terminated, and the corresponding linkage control strategy is formed based on the linkage decision parameters obtained at the time of termination.

[0026] This invention utilizes the time step and the importance contribution sequence of equipment actions to perform fine-weighted decomposition of terminal rewards and generate redistribution rewards. The linkage decision parameters are uniformly trained in the linkage decision output unit that shares the hidden representation with the RUDDER reward decomposition model, so that the linkage control strategy is highly aligned with the actual handling contribution of each device and is more likely to achieve stable convergence and overall performance improvement during reinforcement learning training.

[0027] In this embodiment, the formation of the online event trajectory specifically includes: The linkage control strategy is deployed to the security linkage control execution system. A status acquisition channel for receiving real-time data from each security device is configured, and a strategy invocation interface for sending the current real-time status to the linkage control strategy is established. This allows the linkage control strategy to be periodically invoked during the operation of the security scenario. The security linkage control execution system is a comprehensive control platform used to receive real-time status, invoke the linkage control strategy, and send the linkage control actions output by the strategy to each security device to drive its execution. The real-time status consists of multi-source environmental and device operation data collected from each security device at each time step and processed by time synchronization. When running in a security scenario, the current real-time status is obtained from the status acquisition channel according to the preset time step. The linkage control strategy is input through the strategy call interface. The linkage control strategy generates the linkage control actions of each security device in the current time step, and sends them to the corresponding security devices and drives their execution to complete the linkage control in the current time step. While executing the linkage control actions at each time step, the current real-time status, the actual linkage control actions of each security device issued, and the device feedback information are recorded. The data recorded in the continuous time steps are merged according to the trigger time and end time of the security event to form the corresponding online event trajectory.

[0028] In this embodiment, the implementation of the closed-loop adaptive optimization of the linkage control strategy specifically includes: The online security event trajectories are read one by one, input into the reward decomposition model based on the RUDDER algorithm, and the reward decomposition and reward redistribution are performed to form an online trajectory sample set for updating training. Using the online trajectory sample set as input, the hidden representation generated by the sequence modeling unit is used to iteratively update the internal linkage decision parameters until the preset maximum number of iterations is reached. Based on the updated linkage decision parameters, a new linkage control strategy is formed, realizing closed-loop adaptive optimization of the linkage control strategy.

[0029] Example 1: To verify the feasibility of this invention in practice, it was applied to a comprehensive park security system comprising multiple office buildings, warehouse areas, and perimeter fencing. The park's existing video surveillance, access control, intrusion alarm, perimeter detection, and lighting equipment were already integrated into a traditional rule-based linkage platform. This platform primarily used fixed linkage rules to trigger actions such as "alarm-triggered camera preset position linkage, nighttime-triggered lighting, and access control locking." The maintenance team reported that in complex scenarios, the existing rule-based linkage suffered from high false alarm rates, insufficient timeliness in handling genuine threats, and difficulties in rule maintenance and long-term optimization, especially at night and in key warehouse areas. They desired to improve the accuracy of linkage responses without increasing the workload and energy consumption of security personnel.

[0030] When applying this invention in this scenario, multi-source security data is first continuously collected from the existing security platform in the park, including video summary features from each camera, access control switch records, perimeter detection alarm information, lighting switch status, and security personnel intervention logs. Following the event triggering rules of this invention, the continuous monitoring process is broken down into complete security event trajectories. For example, the process of an abnormal person climbing over the perimeter at night and attempting to enter the warehouse area is divided into continuous time segments from the first trigger of the perimeter detector to the end of the event, forming corresponding security event trajectories. Subsequently, feature encoding is performed on the multi-source data at each time step of each event trajectory to obtain a state sequence; simultaneously, the linkage control commands and execution status of each security device at each time step during the event handling process are organized to form an action sequence. Together, these constitute a trajectory encoding set.

[0031] Based on the park's historical operation records, the operations and maintenance team has set actual handling effect labels for events, such as whether intrusion was successfully prevented, whether manual remediation was required, whether false alarms or missed alarms occurred, and the scope of impact. This invention, based on this, comprehensively evaluates each security event trajectory to obtain a basic evaluation value reflecting the overall handling situation. This value is then weighted according to factors such as regional importance and time-period risk level to form a weighted terminal reward for that trajectory. The event trajectory with weighted terminal rewards is then input into a reward decomposition model built based on the RUDDER algorithm. Through forward sequence modeling and time-step-level reward prediction, a predicted value for the terminal reward of the entire event is given at each time step. The increment of the predicted value between adjacent time steps is then used as the time step contribution. Combined with the embedded representation of different device actions, the time step contribution is further refined to the device action dimension, resulting in a contribution sequence characterizing the importance of time steps and device actions.

[0032] After obtaining the contribution sequence, this invention weights and allocates the terminal rewards according to the proportion of each time step and each device action's contribution to the total contribution of the entire trajectory, generating a redistribution reward sequence refined to the time step and device action. Using the state sequence, action sequence, and corresponding redistribution rewards as training samples, this invention sets up a linkage decision output unit within the reward decomposition model. This unit outputs the predicted results of the linkage control actions of each security device through a shared hidden representation. The redistribution rewards are used to construct the loss, and the linkage decision parameters are iteratively updated, ultimately forming a linkage control strategy that can directly generate multi-device linkage control actions based on the current state.

[0033] During the online operation phase, this invention deploys the linkage control strategy into the park security linkage control execution system. In actual operation, the system collects the current real-time status at fixed time step intervals, inputs it into the linkage control strategy, outputs the linkage control actions of each security device at the current time step, and distributes them to each device for execution via the control module. Simultaneously, the system continuously records the real-time status, strategy output actions, and device feedback information. It merges continuous time step data according to the trigger and end times of security events to form online security event trajectories. These trajectories are then periodically fed into the reward decomposition model to perform reward decomposition, reward reallocation, and linkage decision parameter updates, generating new linkage control strategies and thus forming a closed-loop adaptive optimization process.

[0034] Two consecutive statistical periods were selected in the park as comparison samples. One period used the original traditional rule-based linkage scheme, while the other period used the linkage control strategy of the present invention under the condition that the equipment configuration remained unchanged. The specific comparison data is shown in Table 1. Table 1 Performance comparison data between the present invention and traditional linkage schemes

[0035] As shown in Table 1, under the same park security scenario, the linkage control strategy of this invention demonstrates significant advantages over traditional rule-based linkage schemes in several key indicators: the effective intrusion event handling success rate increases from approximately 87% to approximately 96%, and the average intrusion event handling completion time is shortened from approximately 60 seconds to approximately 38 seconds; the intrusion-related false alarm rate decreases from approximately 12% to approximately 5%, and the proportion of events requiring manual intervention decreases from approximately 22% to approximately 9%; nighttime security-related energy consumption decreases from a relative value of 1.00 to approximately 0.82, and the key equipment action hit rate increases from approximately 73% to approximately 91%. These data indicate that this invention significantly improves response efficiency, reduces false alarms and the frequency of manual intervention while ensuring security handling effectiveness, and also optimizes energy consumption, demonstrating the comprehensive advantages of the reinforcement learning-based linkage control strategy in practical applications.

[0036] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for coordinated control of intelligent security devices based on reinforcement learning, characterized in that, Includes the following steps: Collect security data in the target security area, and break down continuous monitoring into a set of security event trajectories arranged in chronological order according to preset event triggering rules; Feature encoding is performed on the data at each time step in the security event trajectory set, and the linkage behavior of each device is recorded to form a trajectory encoding set; A comprehensive evaluation of the entire trajectory in the trajectory encoding set is performed, and the terminal reward is obtained by combining the scenario risk weighting, resulting in an event trajectory with weighted terminal reward; The event trajectory with weighted terminal feedback is input into the feedback decomposition model based on the RUDDER algorithm. The state sequence and action sequence are sequentially modeled, and the contribution sequence representing the importance of time step and device action is output. The terminal rewards are weighted and allocated according to the contribution sequence to generate a redistribution reward sequence refined to time steps and device actions. At the same time, the linkage decision parameters are jointly trained to form a corresponding linkage control strategy. In a security scenario, the aforementioned linkage control strategy is deployed, taking the real-time status as input, outputting linkage control actions of each device and driving their execution, thus forming an online event trajectory. Based on the decomposition of online security event trajectory execution rewards, the redistribution of rewards, and parameter updates, a closed-loop adaptive optimization of the linkage control strategy is achieved.

2. The method for coordinated control of intelligent security devices based on reinforcement learning according to claim 1, characterized in that, The acquisition of the security event trajectory set specifically includes: Select security devices that participate in the linkage control within the target security area, set device number, physical location, working mode and data reporting cycle for each device, and establish data acquisition channels and time stamping rules with the security platform; The data acquisition channel continuously receives raw security data from each security device at a preset cycle. Data reported by different devices at the same time are aligned and merged according to time stamps to form a multi-source security data time series. Data at each time step is read sequentially from the multi-source security data time series. Preset event triggering rules are called to determine whether the current time step constitutes a security event. Multi-source security data fragments are extracted and merged into a security event trajectory. All trajectories are then sorted to obtain a set of security event trajectories arranged in chronological order.

3. The method for coordinated control of intelligent security devices based on reinforcement learning according to claim 1, characterized in that, The trajectory encoding set specifically includes: For a set of security event trajectories arranged in chronological order, security data from different devices within the same time step are classified and merged according to device identification and time stamp, and organized into a security data time series. For each time step, the security data time series obtained by aggregation is subjected to data cleaning, normalization and format conversion operations. The processed multi-source data is then subjected to feature extraction to generate state feature vectors, which are arranged in chronological order to form a state sequence. While generating the state sequence, the control commands of each security device are read from the linkage control record using the same time step as the state sequence as the index. These commands are then organized and combined to form action vectors. These vectors are arranged in chronological order to form action sequences. The state sequences and action sequences corresponding to each security event trajectory are associated and stored to form a trajectory encoding set.

4. The method for coordinated control of intelligent security devices based on reinforcement learning according to claim 1, characterized in that, The process of obtaining the event trajectory with weighted terminal reports specifically includes: For the trajectory coding set, security event trajectories are read one by one, and trajectory-level evaluation is performed on the entire trajectory. The state change characteristics, equipment action execution and event end status during the event are quantified according to the preset evaluation rules. The quantification results are merged according to the parameter configuration of the evaluation rules to obtain the basic evaluation value. After obtaining the basic evaluation value, the risk level parameters corresponding to the monitoring area and event category of the current event are extracted from the security system configuration. The risk level parameters are used as weighting factors to perform weighted processing on the basic evaluation value, generating a weighted terminal report for the entire security event trajectory. The weighted terminal reports are associated with the state sequence and action sequence one by one, and all records are collected and sorted according to the time of the event to form a set of event trajectories with weighted terminal reports.

5. The method for coordinated control of intelligent security devices based on reinforcement learning according to claim 1, characterized in that, The output of the contribution sequence representing the importance of time steps and equipment actions specifically includes: Based on the set of event trajectories with weighted terminal rewards, the state sequence, action sequence, and weighted terminal reward uniquely corresponding to the entire trajectory are taken as a set of inputs. The reward decomposition model based on the RUDDER algorithm is introduced, and a forward sequence modeling unit that receives state vectors and action vectors in chronological order and outputs forward hidden state sequences and a backward reward propagation unit that propagates weighted terminal reward information backward along time from the end of the trajectory are set up. Within the reward decomposition model, the RUDDER algorithm starts from the initial time step of the security event trajectory. The forward sequence modeling unit receives the state vector and action vector of each time step in chronological order and updates the corresponding forward hidden state at each time step. Meanwhile, the reverse reward propagation unit receives the weighted terminal reward starting from the end of the trajectory and propagates it backward along the time axis. The weighted terminal reward is combined with the forward hidden state and reverse reward information of each time step at the time step level. At each time step, the reward prediction subunit provides the predicted value of the terminal reward of the entire trajectory at the time step truncation and performs differential incremental calculation on the reward prediction results of adjacent time steps to obtain the time step contribution signal. The time step contribution signal, along with the corresponding time step state vector, action vector, and the forward hidden state of the current time step, is input into the device action contribution refinement structure set in the model. By matching the embedded representation of different security devices with the time step hidden state, the device-level weight of each security device is calculated. The time step contribution is divided into different device-level contributions according to the device-level weight. Within each device, based on the feature components of the specific linkage control action and the degree of matching between the feature components and the hidden state, the device-level contribution is further decomposed into each specific control action, resulting in a contribution sequence that represents the dual dimensions of the importance of the time step and the device action.

6. The method for coordinated control of intelligent security devices based on reinforcement learning according to claim 1, characterized in that, The formation of the linkage control strategy specifically includes: Read the importance contribution sequence covering each time step and the actions of each security device, and sum up all the contribution values ​​belonging to the same security event trajectory to obtain the total contribution of the security event trajectory. This total contribution, together with the weighted terminal reward value and contribution sequence of the current trajectory, serves as the basic data set for performing weighted reward allocation. In the aforementioned basic data set, for each security event trajectory, each time step in the trajectory contribution sequence and the corresponding actions of each security device are sequentially traversed. The weighted terminal rewards are weighted and split according to the proportion of the contribution in the total contribution of the trajectory. The reward share obtained from the split is used as the redistribution reward value of the current device action. The redistribution reward values ​​of all time steps and all device actions in the same trajectory are arranged in chronological order and device action order to form a redistribution reward sequence refined to time steps and device actions. In the reward decomposition model based on the RUDDER algorithm, a linkage decision output unit is set up. The state sequence of the trajectory-level sample set is input into the sequence modeling unit in chronological order to generate the hidden representation of each time step. The hidden representation of each time step is sent to the linkage decision output unit to obtain the predicted output of the linkage control action of each security device. The predicted output is matched with the redistribution reward of the corresponding time step and the corresponding device action to construct the loss quantity. After each training iteration, the linkage decision parameters are updated according to the change of the loss quantity. The linkage decision parameters are the hidden representation. When the preset maximum number of iterations is reached, the training is terminated, and the corresponding linkage control strategy is formed based on the linkage decision parameters obtained at the time of termination.

7. The method for coordinated control of intelligent security devices based on reinforcement learning according to claim 1, characterized in that, The formation of the online event trajectory specifically includes: Deploy the linkage control strategy to the security linkage control execution system, configure the status acquisition channel, and establish a strategy call interface to send the current real-time status into the linkage control strategy; When running in a security scenario, the current real-time status is obtained from the status acquisition channel at preset time steps. The linkage control strategy is input through the strategy call interface, and the linkage control strategy generates the linkage control actions of each security device at the current time step, and sends them to the corresponding security devices and drives their execution. While executing the linkage control actions at each time step, the current real-time status, the actual linkage control actions of each security device issued, and the device feedback information are recorded. The data recorded in the continuous time steps are merged according to the trigger time and end time of the security event to form the corresponding online event trajectory.

8. The method for coordinated control of intelligent security devices based on reinforcement learning according to claim 1, characterized in that, The implementation of the closed-loop adaptive optimization of the linkage control strategy specifically includes: The online security event trajectories are read one by one, input into the reward decomposition model based on the RUDDER algorithm, and the reward decomposition and reward redistribution are performed to form an online trajectory sample set for updating training. Using the online trajectory sample set as input, the hidden representation generated by the sequence modeling unit is used to iteratively update the internal linkage decision parameters until the preset maximum number of iterations is reached. Based on the updated linkage decision parameters, a new linkage control strategy is formed, realizing closed-loop adaptive optimization of the linkage control strategy.

Images