Distributed collaborative management and control method based on reinforcement learning

Abstract

The invention discloses a distributed collaborative management and control method based on reinforcement learning, and the method comprises the following steps: obtaining a controlled device set, and configuring an adaptive control unit for each controlled device; the method comprises the following steps: collecting and preprocessing operation state information of controlled equipment, and constructing a local state vector; constructing a joint state vector as a reinforcement learning state; constructing an equipment parameter adjustment action space; constructing a reward evaluation index set, and generating a reinforcement learning interaction sample; on-line strategy updating is executed based on the reinforcement learning interaction sample; generating a target action based on the updated local strategy parameter and the current joint state vector, and issuing and executing the target action; and collecting performance feedback information, triggering Gossip consistency propagation to correct local strategy parameters, and generating a dynamic collaborative optimization result and a strategy consistency control result. According to the method, a reinforcement learning and Gossip cooperation mechanism is adopted, self-adaptive management and control of the distributed equipment are realized, and the method has the advantages of high stability and fast cooperation convergence.

Description

Technical Field

[0001] This invention relates to the field of equipment collaborative control technology, and in particular to a distributed collaborative management and control method based on reinforcement learning. Background Technology

[0002] Existing equipment management systems mostly adopt centralized control architectures or distributed control methods based on fixed rules, adjusting equipment operating parameters through preset thresholds, parameter tables, or manual experience models. In multi-device collaborative scenarios, some solutions introduce simple information exchange mechanisms, achieving parameter synchronization through periodic broadcasting of operating status or using consistency protocols. However, most still rely on static rules or offline modeling methods for decision-making, lacking adaptive learning capabilities based on real-time operating status, making it difficult to achieve continuous optimization under dynamic load changes or complex operating conditions.

[0003] In the aforementioned technical solutions, due to the lack of joint modeling and reinforcement learning policy update mechanisms for local and neighbor states, inter-device collaborative control often suffers from response lag, parameter oscillations, or difficulty in achieving global performance convergence. Existing consistency processing methods typically lack explicit deviation metrics and weighted aggregation rules, resulting in a lack of deterministic constraints in the consistency correction process. This makes it impossible to simultaneously consider global indicators, resource contention, and security boundary requirements, thus limiting system stability and collaborative optimization effectiveness.

[0004] Therefore, how to provide a distributed collaborative management and control method based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a distributed collaborative management and control method based on reinforcement learning. This invention employs a reinforcement learning and Gossip collaborative mechanism to achieve adaptive management and control of distributed devices, which has the advantages of high stability and fast collaborative convergence.

[0006] A distributed collaborative management and control method based on reinforcement learning according to an embodiment of the present invention includes the following steps: Obtain the set of controlled devices and configure the corresponding adaptation control unit for each controlled device in the set; Each adapter control unit collects the operating status information of the corresponding controlled device, performs preprocessing on the operating status information, obtains a standardized operating status sequence, and constructs a local state vector; The local state vector is concatenated with the neighbor state summary vector received through the communication link to construct a joint state vector, which is then used as the reinforcement learning state. Construct the action space for adjusting equipment parameters and set the amplitude constraints, frequency constraints, and mapping relationships of parameter adjustment commands; Construct a set of reward evaluation metrics, calculate reward values ​​based on local state vectors and neighbor state summary vectors, and generate reinforcement learning interaction samples; The updated local policy parameters are obtained by performing online policy updates based on reinforcement learning interaction samples, and collaborative constraint information is introduced during the update process. The target action is generated based on the updated local policy parameters and the current joint state vector and then sent out for execution via the parameter read / write channel. Each adapter control unit collects performance feedback information after the controlled device's execution parameters are adjusted, and triggers Gossip consistency propagation to correct local policy parameters based on consistency deviation measurement, generating dynamic collaborative optimization results and policy consistency control results.

[0007] Optionally, the adapter control unit establishes a parameter read / write channel with the corresponding controlled device, and establishes a communication link between the adapter control units and completes node identifier configuration.

[0008] Optionally, the preprocessing includes timestamp alignment, outlier handling, missing value handling, dimension normalization, and unified sampling frequency processing. The construction of the local state vector includes: parsing and extracting features based on a standardized operating state sequence according to a preset set of state indicators, mapping electrical parameter features, load parameter features, temperature parameter features, alarm identifier features, and control response features to their corresponding feature dimensions; aligning and aggregating multi-dimensional features at the same time step according to a unified sampling time window; performing numerical normalization mapping processing on continuous features, enumeration encoding processing on discrete features, and binarization marking processing on alarm features; vectorizing and concatenating the features of each dimension according to the feature index order to form a fixed-dimensional operating state feature vector; and performing time window cascaded update processing on the operating state feature vector to introduce time continuity features and obtain the local state vector.

[0009] Optionally, the construction of the joint state vector specifically includes: Each adaptation control unit determines the mapping rule between the set of reinforcement learning state indicators and the state dimension based on the local state vector, and configures an indicator index for each state indicator in the set of reinforcement learning state indicators. Based on the node identifier configuration result, a local state vector summary generation request is sent to the communication neighbor through the communication link, and the neighbor state summary vector returned by the communication neighbor is received. The neighbor state summary vector is obtained by the communication neighbor performing dimensionality reduction and aggregation processing on the local state vector. Perform consistency verification and time alignment on the received neighbor state digest vector to obtain the aligned neighbor state digest vector; Based on the state dimension mapping rule, a combination operation is performed on the local state vector and the aligned neighbor state summary vector to generate a joint state vector, which is used as the reinforcement learning state in the reinforcement learning state space.

[0010] Optionally, the construction of the device parameter adjustment action space specifically includes: A parameter index table is established based on the set of operating parameters of the corresponding controlled equipment, a subset of adjustable operating parameters is determined, and parameter identifiers and parameter read / write addresses are configured for each operating parameter in the subset of adjustable operating parameters. The device parameter adjustment action space is constructed based on the parameter index table. The device parameter adjustment action space is defined as a discrete action set consisting of a parameter adjustment instruction set. The parameter adjustment instruction set includes instruction types that perform upward adjustment, downward adjustment and hold operations on each operating parameter respectively. Each adapter control unit sets amplitude and frequency constraints for parameter adjustment commands. Establish a mapping relationship between parameter adjustment commands and parameter read / write channels, and form a parameter write message structure based on the parameter read / write address.

[0011] Optionally, the generation of the reinforcement learning interaction samples specifically includes: A set of reward evaluation indicators is constructed based on the local state vector and the neighbor state summary vector received through the communication link, and indicator weights are configured for each reward evaluation indicator in the set of reward evaluation indicators. The performance metrics, parameter stability metrics, and resource consumption metrics are calculated based on local state vectors, and the collaborative consistency metrics are calculated based on neighbor state summary vectors. The reward value is calculated based on the set of reward evaluation indicators. The reward value is the weighted sum of the operation performance indicator and the parameter stability indicator minus the weighted sum of the coordination consistency indicator and the resource consumption indicator. Each adaptation control unit associates and encapsulates the current joint state vector, the target action and reward value at the current time step into an interaction record. At the next sampling time, it generates the next-time joint state vector based on the updated local state vector and the updated neighbor state summary vector. The interaction record and the next-time joint state vector are combined to generate reinforcement learning interaction samples.

[0012] Optionally, obtaining the updated local policy parameters specifically includes: Obtain a set of reinforcement learning interaction samples associated with local node identifiers, and construct an online policy update batch based on the set of reinforcement learning interaction samples; The strategy improvement target value is calculated based on the online strategy update batch. The strategy improvement target value is the weighted combination of the time cumulative weighted result of the reward value sequence and the estimated return value corresponding to the joint state vector sequence at the next time step. In the process of calculating the target value of strategy improvement, collaborative constraint information is introduced. The collaborative constraint information is converted into constraint penalty value and combined with the target value of strategy improvement in a weighted manner to obtain the constraint update target value. The constraint penalty value is the weighted sum of the target global indicator constraint penalty value, resource competition constraint penalty value, strategy consistency constraint penalty value and security boundary constraint penalty value. The policy parameter update amount is calculated based on the constraint update target value. The policy parameter update amount is the product of the gradient direction of the constraint update target value with respect to the local policy parameter and the preset learning rate. The updated local policy parameters are obtained by applying the policy parameter update amount to the local policy parameters, and then the updated local policy parameters are written into the policy parameter storage structure.

[0013] Optionally, the execution of the delivery specifically includes: Obtain the current joint state vector and read the updated local policy parameters. Based on the updated local policy parameters, perform action selection calculation on the current joint state vector to obtain the target action. Perform constraint verification on the target action to obtain the constraint verification result; When the constraint verification result indicates that the amplitude constraint and frequency constraint are satisfied, the target action is parsed into the target running parameter identifier, the target parameter value and the effective time identifier according to the mapping relationship between the parameter adjustment instruction and the parameter read and write channel, and a parameter write message is formed based on the parameter read and write address; Based on the parameter write message, the target action is sent to the corresponding controlled device through the parameter read and write channel. On the controlled device side, the target parameter value is written to the target operating parameter according to the target operating parameter identifier. At the effective time corresponding to the effective time identifier, the operating parameter set update operation is performed and the operating parameter adjustment process is triggered.

[0014] Optionally, the generation of the dynamic collaborative optimization results and the policy consistency control results specifically includes: After the target action is sent through the parameter read / write channel, the performance feedback information of the corresponding controlled device is collected, and a feedback summary is generated based on the performance feedback information. The communication neighbor feedback summary is received through the communication link, and a consistency deviation metric is calculated based on the local feedback summary and the communication neighbor feedback summary. The consistency deviation metric is the weighted sum of the deviations between the local feedback summary and the communication neighbor feedback summary on key feedback indicators. The consistency deviation metric is compared with the consistency threshold. When the consistency deviation metric exceeds the consistency threshold, the Gossip protocol consistency propagation process is initiated. When the consistency deviation metric does not exceed the consistency threshold, the feedback digest collection and consistency deviation metric calculation are maintained. In the Gossip protocol consensus propagation process, a fixed number of propagation operations are performed sequentially according to the communication cycle. In each propagation operation, a communication neighbor is determined based on the adjacency topology, feedback digests are exchanged with the communication neighbor, and a weighted aggregation operation is performed on the exchanged feedback digests to update the consensus correction digest. After completing the fixed number of propagation operations, the consensus correction digest is output. Perform consistency correction updates on local policy parameters based on the consistency correction summary; The dynamic collaborative optimization result is obtained by calculating the policy output of the current joint state vector based on the corrected local policy parameters. The convergence degree of the consistency deviation is calculated based on the consistency deviation metric and the policy parameter correction amount. The policy consistency control result is generated by hierarchical judgment based on the convergence degree of the consistency deviation.

[0015] The beneficial effects of this invention are: This invention constructs a distributed collaborative management and control architecture based on reinforcement learning. It combines local state vectors with neighbor state summary vectors to form a joint state vector, which serves as the reinforcement learning state, enabling global modeling and dynamic perception of the operational status of multiple devices. By constructing a device parameter adjustment action space and setting amplitude constraints, frequency constraints, and parameter read / write channel mapping relationships, the target actions output by the policy can be accurately executed within safe boundaries, avoiding the oscillation phenomenon caused by parameter mutations or frequent adjustments in traditional rule-based control. In the reward evaluation mechanism, a weighted calculation method is introduced using operational performance indicators, parameter stability indicators, collaborative consistency indicators, and resource consumption indicators. This allows the policy update process to simultaneously consider local performance and global collaborative effects, thereby achieving continuous optimization under dynamic loads and complex operating conditions.

[0016] Meanwhile, this application constructs a Gossip consistency propagation mechanism that uses a consistency deviation metric and a threshold-triggered approach. When the deviation exceeds the threshold, weighted aggregation is performed to generate a consistency correction summary. Based on this summary, local policy parameters are deterministically corrected and updated, gradually bringing policy parameters across multiple nodes closer to consistency and improving the overall convergence and stability of the system. This consistency correction mechanism, combined with collaborative constraint information, achieves comprehensive constraint control over global indicators, resource contention, and security boundaries, enhancing multi-device collaborative optimization capabilities while ensuring safe device operation.

[0017] Through the synergistic effect of the aforementioned joint state modeling, reinforcement learning online update, and consistency correction propagation mechanism, this application can achieve adaptive parameter adjustment and policy consistency control in a distributed environment, reduce the dependence of centralized control on system stability, improve compatibility and scalability for different types of equipment, and thus significantly enhance the intelligence level and operational reliability of multi-device collaborative management and control. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a distributed collaborative management and control method based on reinforcement learning proposed in this invention; Figure 2 This is a flowchart illustrating the online policy update process of a distributed collaborative management and control method based on reinforcement learning proposed in this invention. Figure 3 This is a flowchart illustrating the Gossip consensus propagation process of a distributed collaborative management method based on reinforcement learning proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figures 1-3 A distributed collaborative management and control method based on reinforcement learning includes the following steps: Obtain the set of controlled devices and configure the corresponding adaptation control unit for each controlled device in the set; Each adapter control unit collects the operating status information of the corresponding controlled device, performs preprocessing on the operating status information, obtains a standardized operating status sequence, and constructs a local state vector; The local state vector is concatenated with the neighbor state summary vector received through the communication link to construct a joint state vector, which is then used as the reinforcement learning state. Construct the action space for adjusting equipment parameters and set the amplitude constraints, frequency constraints, and mapping relationships of parameter adjustment commands; Construct a set of reward evaluation metrics, calculate reward values ​​based on local state vectors and neighbor state summary vectors, and generate reinforcement learning interaction samples; The updated local policy parameters are obtained by performing online policy updates based on reinforcement learning interaction samples, and collaborative constraint information is introduced during the update process. The target action is generated based on the updated local policy parameters and the current joint state vector and then sent out for execution via the parameter read / write channel. Each adapter control unit collects performance feedback information after the controlled device's execution parameters are adjusted, and triggers Gossip consistency propagation to correct local policy parameters based on consistency deviation measurement, generating dynamic collaborative optimization results and policy consistency control results.

[0021] In this embodiment, the adapter control unit establishes a parameter read / write channel with the corresponding controlled device, and establishes a communication link between the adapter control units and completes the node identifier configuration.

[0022] In this embodiment, preprocessing includes timestamp alignment, outlier handling, missing value handling, dimension normalization, and unified sampling frequency processing. The construction of the local state vector includes: parsing and extracting features based on a standardized operating state sequence according to a preset set of state indicators, mapping electrical parameter features, load parameter features, temperature parameter features, alarm identifier features, and control response features to their corresponding feature dimensions; aligning and aggregating multi-dimensional features at the same time step according to a unified sampling time window; performing numerical normalization mapping processing on continuous features, enumeration encoding processing on discrete features, and binarization marking processing on alarm features; vectorizing and concatenating the features of each dimension according to the feature index order to form a fixed-dimensional operating state feature vector; and performing time window cascaded update processing on the operating state feature vector to introduce time continuity features and obtain the local state vector.

[0023] In this embodiment, the construction of the joint state vector specifically includes: Each adaptation control unit determines the set of reinforcement learning state indicators and the state dimension mapping rule based on the local state vector, and configures an indicator index for each state indicator in the set of reinforcement learning state indicators. The set of reinforcement learning state indicators includes the operating parameter features, performance feedback features, constraint occupancy features and neighbor collaboration features of the corresponding controlled device. The state dimension mapping rule is used to limit the dimensional position and splicing order of the local state vector and the neighbor state summary vector in the joint state vector. Based on the node identifier configuration result, a local state vector summary generation request is sent to the communication neighbor through the communication link, and the neighbor state summary vector returned by the communication neighbor is received. The neighbor state summary vector is obtained by the communication neighbor performing dimensionality reduction and aggregation processing on the local state vector. The dimensionality reduction and aggregation processing includes grouping the local state vector according to the index and calculating the statistical characteristics within the group respectively. The statistical characteristics within the group include the mean characteristic within the group, the fluctuation range characteristic within the group, and the trend characteristic within the group. The received neighbor state digest vector is subjected to consistency verification and time alignment processing to obtain an aligned neighbor state digest vector. The consistency verification includes verifying the consistency of the neighbor source based on the node identifier and verifying the timeliness consistency of the neighbor state digest vector based on the timestamp. The time alignment processing includes mapping the neighbor state digest vector to the same sampling time as the local state vector and performing completion processing on the neighbor state digest vector with missing sampling time. According to the state dimension mapping rule, a combination operation is performed on the local state vector and the aligned neighbor state summary vector to generate a joint state vector, which serves as the reinforcement learning state in the reinforcement learning state space. The combination operation includes placing the local state vector in the local state segment of the joint state vector according to the index order, and placing the aligned neighbor state summary vector in the neighbor state segment of the joint state vector according to the node identifier order. The dimension length of the joint state vector is the sum of the dimension length of the local state vector and the dimension length of each aligned neighbor state summary vector.

[0024] In this embodiment, the construction of the device parameter adjustment action space specifically includes: A parameter index table is established based on the set of operating parameters of the corresponding controlled equipment. A subset of adjustable operating parameters is determined, and parameter identifiers and parameter read / write addresses are configured for each operating parameter in the subset of adjustable operating parameters. The subset of adjustable operating parameters includes voltage parameters, output current parameters, power limiting parameters, temperature protection threshold parameters, and start / stop control parameters. The device parameter adjustment action space is constructed based on the parameter index table. The device parameter adjustment action space is defined as a discrete action set consisting of a set of parameter adjustment instructions. The set of parameter adjustment instructions includes instruction types that perform up adjustment, down adjustment, and hold operations on each operating parameter. Any parameter adjustment instruction in the discrete action set is used as a candidate action for reinforcement learning. Up adjustment operation corresponds to increasing the current parameter value of the target operating parameter by a preset adjustment step size. Down adjustment operation corresponds to decreasing the current parameter value of the target operating parameter by a preset adjustment step size. Hold operation corresponds to keeping the current parameter value of the target operating parameter unchanged. Each adapter control unit sets amplitude and frequency constraints for parameter adjustment commands. Amplitude constraints include configuring minimum and maximum allowable values ​​for each operating parameter, and when the target action is subsequently converted into a parameter adjustment command, limiting the target parameter value for upward adjustment to within the maximum allowable value and the target parameter value for downward adjustment to above the minimum allowable value. Frequency constraints include configuring a minimum adjustment interval for each operating parameter and recording the timestamp of the most recent parameter adjustment command, and when the target action is subsequently converted into a parameter adjustment command, determining whether the minimum adjustment interval is met based on the time difference between the current timestamp and the timestamp of the command. Establish a mapping relationship between parameter adjustment commands and parameter read / write channels, and form a parameter write message structure based on the parameter read / write address. The mapping relationship includes parsing the parameter adjustment command into a target running parameter identifier, a target parameter value, and an effective time identifier. The parameter write message structure is used to subsequently send the parameter to the corresponding controlled device through the parameter read / write channel.

[0025] In this embodiment, the generation of reinforcement learning interaction samples specifically includes: A set of reward evaluation indicators is constructed based on the local state vector and the neighbor state summary vector received through the communication link, and indicator weights are configured for each reward evaluation indicator in the set of reward evaluation indicators. The set of reward evaluation indicators includes operation performance indicators, parameter stability indicators, collaborative consistency indicators and resource consumption indicators. The operating performance index, parameter stability index, and resource consumption index are calculated based on the local state vector. The collaborative consistency index is calculated based on the neighbor state summary vector. The operating performance index represents the degree to which the output performance of the controlled device meets the preset target range at the current sampling time. The parameter stability index represents the change range of the set of operating parameters of the controlled device determined by the difference between the local state vector at the current sampling time and the local state vector at the previous sampling time. The resource consumption index represents the level of the controlled device's occupation of shared resources. The collaborative consistency index represents the deviation level between the local state vector and the neighbor state summary vector on key operating indicators. The reward value is calculated based on the set of reward evaluation indicators. The reward value is the weighted sum of the operation performance indicators and parameter stability indicators minus the weighted sum of the coordination consistency indicators and resource consumption indicators. The weighted sum is the value obtained by multiplying each reward evaluation indicator by its corresponding weight and then summing the results. The indicator weight is used to characterize the degree of contribution of the corresponding reward evaluation indicator to the reward value. Each adaptation control unit associates and encapsulates the current joint state vector, the target action at the current time step, and the reward value into an interaction record. At the next sampling time, it generates the next-time joint state vector based on the updated local state vector and the updated neighbor state summary vector. The interaction record and the next-time joint state vector are combined to generate reinforcement learning interaction samples. The target action is the parameter adjustment instruction in the device parameter adjustment action space.

[0026] In this embodiment, obtaining the updated local policy parameters specifically includes: Obtain a set of reinforcement learning interaction samples associated with local node identifiers, and construct an online policy update batch based on the set of reinforcement learning interaction samples. The online policy update batch includes the current joint state vector sequence, the target action sequence at the current time step, the reward value sequence, and the joint state vector sequence at the next time step. The strategy improvement target value is calculated based on the online policy update batch. The strategy improvement target value is the weighted combination of the time-cumulative weighted result of the reward value sequence and the estimated return value corresponding to the joint state vector sequence at the next time step. The time-cumulative weighted result is the value obtained by weighting and accumulating the reward values ​​at each time step according to the time decay weight. The estimated return value is the value obtained by evaluating the return of the joint state vector at the next time step based on the current local policy parameters. In the process of calculating the target value of policy improvement, collaborative constraint information is introduced. The collaborative constraint information is converted into constraint penalty value and weighted and combined with the target value of policy improvement to obtain the constraint update target value. The constraint penalty value is the weighted sum of the target global indicator constraint penalty value, resource contention constraint penalty value, policy consistency constraint penalty value and security boundary constraint penalty value. Among them, the target global indicator constraint penalty value represents the degree to which the state component reflecting the global operation indicator in the joint state vector deviates from the preset target range; the resource contention constraint penalty value represents the degree to which the state component reflecting the shared resource occupation in the joint state vector exceeds the preset resource threshold; the policy consistency constraint penalty value represents the degree to which the difference between the communication neighbor policy parameter summary received through the communication link and the local policy parameter is exceeded; and the security boundary constraint penalty value represents the degree to which the parameter adjustment instruction corresponding to the target action triggers the amplitude constraint or frequency constraint violation. The policy parameter update amount is calculated based on the constraint update target value. The policy parameter update amount is the product of the gradient direction of the constraint update target value with respect to the local policy parameter and the preset learning rate. The gradient direction represents the adjustment direction of the local policy parameter determined based on the constraint update target value, and the preset learning rate represents the parameter adjustment step size for each online policy update. The updated local policy parameters are obtained by applying the policy parameter update amount to the local policy parameters, and then the updated local policy parameters are written into the policy parameter storage structure.

[0027] In this embodiment, the execution process specifically includes: Obtain the current joint state vector and read the updated local policy parameters. Perform action selection calculation on the current joint state vector based on the updated local policy parameters to obtain the target action, which is the parameter adjustment instruction in the device parameter adjustment action space. The constraint verification process is performed on the target action to obtain the constraint verification result. The constraint verification process includes verifying whether the target parameter value corresponding to the target action is between the minimum allowable value and the maximum allowable value of the parameter based on the amplitude constraint, and verifying whether the time difference between the current timestamp and the timestamp of the most recent parameter adjustment instruction meets the minimum adjustment interval duration based on the frequency constraint. When the constraint verification result indicates that the amplitude constraint and frequency constraint are satisfied, the target action is parsed into the target running parameter identifier, the target parameter value and the effective time identifier according to the mapping relationship between the parameter adjustment instruction and the parameter read and write channel, and a parameter write message is formed based on the parameter read and write address; Based on the parameter write message, the target action is sent to the corresponding controlled device through the parameter read and write channel. On the controlled device side, the target parameter value is written to the target operating parameter according to the target operating parameter identifier. At the effective time corresponding to the effective time identifier, the operating parameter set update operation is performed and the operating parameter adjustment process is triggered.

[0028] In this embodiment, the generation of dynamic collaborative optimization results and policy consistency control results specifically includes: After the target action is sent through the parameter read / write channel, the performance feedback information of the corresponding controlled device is collected, and a feedback summary is generated based on the performance feedback information. The performance feedback information includes output voltage feedback, output current feedback, power feedback, temperature feedback and alarm status feedback. The feedback summary includes a feedback index vector arranged in the order of the preset feedback index and the node identifier and feedback timestamp associated with the feedback index vector. The system receives feedback summaries from communication neighbors via a communication link and calculates a consistency deviation metric based on the local feedback summary and the communication neighbor feedback summary. The consistency deviation metric is a weighted sum of the deviations between the local feedback summary and the communication neighbor feedback summary on key feedback indicators. The deviation is the absolute value of the numerical difference between the corresponding key feedback indicators. The weighted sum of deviations is the sum of the deviations of each key feedback indicator multiplied by its corresponding deviation weight. The deviation weight is used to characterize the contribution of the corresponding key feedback indicator to the consistency deviation metric. The consistency deviation metric is compared with the consistency threshold. When the consistency deviation metric exceeds the consistency threshold, the Gossip protocol consistency propagation process is initiated. When the consistency deviation metric does not exceed the consistency threshold, the feedback digest collection and consistency deviation metric calculation are maintained. In the Gossip protocol consensus propagation process, a fixed number of propagation operations are performed sequentially according to the communication cycle. In each propagation operation, a communication neighbor is determined based on the adjacency topology, feedback digests are exchanged with the communication neighbor, and a weighted aggregation operation is performed on the exchanged feedback digests to update the consensus correction digest. After completing the fixed number of propagation operations, the consensus correction digest is output. The weighted aggregation operation is a digest vector obtained by weighting the local feedback digest and the exchanged feedback digest according to a preset aggregation weight. The aggregation weight is used to characterize the proportion of the local feedback digest and the exchanged feedback digest in the consensus correction digest. The local policy parameters are updated based on the consistency correction summary. The consistency correction update includes determining the policy parameter correction amount based on the difference between the consistency correction summary and the local policy parameters, and applying the policy parameter correction amount to the local policy parameters to obtain the corrected local policy parameters. The dynamic collaborative optimization result is obtained by calculating the policy output of the current joint state vector based on the corrected local policy parameters. The convergence degree of the consistency deviation is calculated based on the consistency deviation metric and the policy parameter correction amount. The policy consistency control result is generated by hierarchical judgment based on the convergence degree of the consistency deviation. The policy output calculation includes: constructing a policy output function based on the corrected local policy parameters; taking the current joint state vector as input data and performing vector expansion processing according to the preset state index order; inputting the expanded joint state vector into the policy output function to perform policy score calculation to obtain the policy score value corresponding to each candidate parameter adjustment instruction; performing comparison and sorting processing on the policy score values ​​of each candidate parameter adjustment instruction; selecting the parameter adjustment instruction with the highest score as the target action output; and associating and encapsulating the target action with the current joint state vector and the corresponding policy score value to generate dynamic collaborative optimization results. The classification determination includes: calculating the change in consistency deviation based on the consistency deviation metric and the policy parameter correction amount; performing a difference operation between the consistency deviation metric at the current sampling time and the consistency deviation metric at the previous sampling time to obtain the change in deviation value; and weighting the change in deviation value with the magnitude value of the policy parameter correction amount to obtain the degree of convergence of consistency deviation; performing interval matching processing based on the degree of convergence of consistency deviation and a preset set of classification interval thresholds to determine the consistency level identifier; generating a corresponding control identifier code based on the consistency level identifier; associating and encapsulating the consistency level identifier and the control identifier code to generate the policy consistency control result.

[0029] Example 1: To verify the feasibility of this invention in practice, it was applied to a distributed power supply and energy storage coordinated control system in an industrial park. The system consists of multiple grid-connected power supply units, energy storage units, and load units. Each unit is equipped with an adaptive control unit and forms a coordinated network through a communication link. In traditional control methods, each device often adopts independent adjustment or centralized control mode, which easily leads to problems such as response lag, effective local adjustment but large fluctuations in global indicators. Especially under conditions of sudden load changes or power fluctuations, there are issues such as parameter oscillations, frequent adjustments, and inconsistent control between devices.

[0030] In this scenario, the present invention configures an adaptive control unit for each controlled device, collects operational status information in real time, constructs a local state vector, and then combines it with the neighbor state summary vector returned by the communicating neighbor to construct a joint state vector, which is used as the state input for reinforcement learning. Based on the constructed device parameter adjustment action space and reward evaluation index set, the system continuously generates reinforcement learning interaction samples and performs online policy updates, enabling local policy parameters to adaptively adjust with changes in the operating environment. Once a target action is generated, it is sent to the corresponding controlled device for execution through the parameter read / write channel, thereby achieving dynamic adjustment of operating parameters such as voltage, current, power limits, and temperature protection thresholds.

[0031] During operation, each adaptation control unit continuously collects performance feedback information from the controlled devices after parameter adjustments, generates feedback summaries, and calculates a consistency deviation metric. When the consistency deviation metric exceeds the consistency threshold, the Gossip consistency propagation process is automatically triggered. In a fixed number of propagation rounds, feedback summaries are exchanged with communication neighbors, and weighted aggregation operations are performed to generate a consistency correction summary. Based on this summary, the local policy parameters are updated to reflect consistency correction. Subsequently, based on the corrected local policy parameters, the policy output is calculated for the current joint state vector, generating a dynamic collaborative optimization result. Simultaneously, the consistency deviation convergence degree is calculated by combining the consistency deviation metric and the policy parameter correction amount. A hierarchical determination is then used to generate the policy consistency control result.

[0032] Through the above implementation methods, the system can achieve closed-loop control of adaptive policy updates and consistency correction in scenarios with load fluctuations and multi-node collaborative adjustment. This avoids global index imbalance caused by single-point optimization, improves the degree of collaboration among distributed devices, reduces the risk of parameter oscillation, and achieves stable, coordinated, and sustainable distributed collaborative management and control.

[0033] Table 1. Performance Comparison of the Invention and Traditional Centralized Control Methods

[0034] As can be clearly seen from Table 1, the method of the present invention is superior to the traditional method in many indicators.

[0035] In terms of voltage fluctuation amplitude, the traditional method results in 4.8V, while the method of this invention results in 3.6V, a reduction of 1.2V. This reduction in voltage fluctuation amplitude indicates more coordinated adjustment among multiple nodes. This invention constructs a joint state vector and introduces a cooperative consistency index into the reward function, enabling each adaptation control unit to comprehensively consider the neighbor's state summary vector when performing parameter adjustments. This avoids voltage reversal caused by overcompensation of a single node, thus effectively suppressing the fluctuation amplitude.

[0036] Regarding frequency deviation, the traditional method achieves 0.18Hz, while the method of this invention achieves 0.11Hz, a reduction of 0.07Hz. This improvement stems from the introduction of an online policy update mechanism and collaborative constraint information. This allows the policy improvement target value to not only focus on local rewards but also suppress deviations from global operating indicators through constraint penalty values. Therefore, it can approach the rated frequency more quickly in scenarios with fluctuating loads across multiple devices.

[0037] The steady-state recovery time was reduced from 12.5s to 9.2s, a decrease of 3.3s. This reduction in recovery time is related to the continuous generation of reinforcement learning interaction samples and the real-time updating of policy parameters. This invention immediately collects performance feedback information after executing the target action and triggers the Gossip consensus propagation process when the consensus deviation exceeds the consensus threshold, thereby achieving consensus correction and updating of policy parameters. This allows the system to reach a new equilibrium state more quickly after disturbances.

[0038] The number of parameter adjustments decreased from 42 times / hour to 31 times / hour, a reduction of 11 times / hour. This decrease indicates a smoother control process. This effect stems from the amplitude and frequency constraints set in the action space, as well as the parameter stability index in the reward function. These constraints cause the policy output function to tend to select parameter adjustment instructions with better stability during action selection calculations, thereby reducing frequent adjustments.

[0039] The consensus deviation metric decreased from 0.085 pu to 0.062 pu, a difference of 0.023 pu. This metric directly reflects the consistency of operational metrics among multiple nodes. This invention generates a consensus correction digest through weighted aggregation operations and performs consensus correction updates on local policy parameters based on the consensus correction digest, effectively reducing the difference between the local feedback digest and the feedback digest of the communicating neighbors, thus significantly reducing the consensus deviation.

[0040] Resource utilization rate decreased from 78% to 72%, a drop of 6%. This is because the reward evaluation index set includes resource utilization indicators, and the resource competition constraint penalty value is weighted in the strategy improvement target value, which suppresses high resource utilization behavior during the strategy parameter update process and achieves a more balanced resource allocation.

[0041] The control success rate improved from 94.6% to 96.3%, a 1.7% increase. This improvement stems from the hierarchical judgment mechanism of the strategy consistency control results. By calculating the convergence degree of consistency deviation and generating control identifier codes, response controls are applied to different consistency levels, thereby improving overall control stability and execution reliability. Overall data shows that this invention achieves robust improvements in stability, consistency, and resource utilization efficiency.

[0042] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A distributed collaborative management and control method based on reinforcement learning, characterized in that, Includes the following steps: Obtain the set of controlled devices and configure the corresponding adaptation control unit for each controlled device in the set; Each adapter control unit collects the operating status information of the corresponding controlled device, performs preprocessing on the operating status information, obtains a standardized operating status sequence, and constructs a local state vector; The local state vector is concatenated with the neighbor state summary vector received through the communication link to construct a joint state vector, which is then used as the reinforcement learning state. Construct the action space for adjusting equipment parameters and set the amplitude constraints, frequency constraints, and mapping relationships of parameter adjustment commands; Construct a set of reward evaluation metrics, calculate reward values ​​based on local state vectors and neighbor state summary vectors, and generate reinforcement learning interaction samples; The updated local policy parameters are obtained by performing online policy updates based on reinforcement learning interaction samples, and collaborative constraint information is introduced during the update process. The target action is generated based on the updated local policy parameters and the current joint state vector and then sent out for execution via the parameter read / write channel. Each adapter control unit collects performance feedback information after the controlled device's execution parameters are adjusted, and triggers Gossip consistency propagation to correct local policy parameters based on consistency deviation measurement, generating dynamic collaborative optimization results and policy consistency control results.

2. The distributed collaborative management and control method based on reinforcement learning according to claim 1, characterized in that, The adapter control unit establishes a parameter read / write channel with the corresponding controlled device, and establishes a communication link between the adapter control units and completes the node identifier configuration.

3. The distributed collaborative management and control method based on reinforcement learning according to claim 1, characterized in that, The preprocessing includes timestamp alignment, outlier handling, missing value handling, dimensional normalization, and unified sampling frequency processing. The construction of the local state vector includes: parsing and extracting features based on a standardized operating state sequence according to a preset set of state indicators, mapping electrical parameter features, load parameter features, temperature parameter features, alarm identifier features, and control response features to their corresponding feature dimensions; aligning and aggregating multi-dimensional features at the same time step according to a unified sampling time window; performing numerical normalization mapping on continuous features, enumeration encoding on discrete features, and binarization marking on alarm features; vectorizing and concatenating the features according to the feature index order to form a fixed-dimensional operating state feature vector; and performing time window cascaded update processing on the operating state feature vector to introduce time continuity features and obtain the local state vector.

4. The distributed collaborative management and control method based on reinforcement learning according to claim 1, characterized in that, The construction of the joint state vector specifically includes: Each adaptation control unit determines the mapping rule between the set of reinforcement learning state indicators and the state dimension based on the local state vector, and configures an indicator index for each state indicator in the set of reinforcement learning state indicators. Based on the node identifier configuration result, a local state vector summary generation request is sent to the communication neighbor through the communication link, and the neighbor state summary vector returned by the communication neighbor is received. The neighbor state summary vector is obtained by the communication neighbor performing dimensionality reduction and aggregation processing on the local state vector. Perform consistency verification and time alignment on the received neighbor state digest vector to obtain the aligned neighbor state digest vector; Based on the state dimension mapping rule, a combination operation is performed on the local state vector and the aligned neighbor state summary vector to generate a joint state vector, which is used as the reinforcement learning state in the reinforcement learning state space.

5. The distributed collaborative management and control method based on reinforcement learning according to claim 1, characterized in that, The construction of the device parameter adjustment action space specifically includes: A parameter index table is established based on the set of operating parameters of the corresponding controlled equipment, a subset of adjustable operating parameters is determined, and parameter identifiers and parameter read / write addresses are configured for each operating parameter in the subset of adjustable operating parameters. The device parameter adjustment action space is constructed based on the parameter index table. The device parameter adjustment action space is defined as a discrete action set consisting of a parameter adjustment instruction set. The parameter adjustment instruction set includes instruction types that perform upward adjustment, downward adjustment and hold operations on each operating parameter respectively. Each adapter control unit sets amplitude and frequency constraints for parameter adjustment commands. Establish a mapping relationship between parameter adjustment commands and parameter read / write channels, and form a parameter write message structure based on the parameter read / write address.

6. The distributed collaborative management and control method based on reinforcement learning according to claim 1, characterized in that, The generation of the reinforcement learning interaction samples specifically includes: A set of reward evaluation indicators is constructed based on the local state vector and the neighbor state summary vector received through the communication link, and indicator weights are configured for each reward evaluation indicator in the set of reward evaluation indicators. The performance metrics, parameter stability metrics, and resource consumption metrics are calculated based on local state vectors, and the collaborative consistency metrics are calculated based on neighbor state summary vectors. The reward value is calculated based on the set of reward evaluation indicators. The reward value is the weighted sum of the operation performance indicator and the parameter stability indicator minus the weighted sum of the coordination consistency indicator and the resource consumption indicator. Each adaptation control unit associates and encapsulates the current joint state vector, the target action and reward value at the current time step into an interaction record. At the next sampling time, it generates the next-time joint state vector based on the updated local state vector and the updated neighbor state summary vector. The interaction record and the next-time joint state vector are combined to generate reinforcement learning interaction samples.

7. The distributed collaborative management and control method based on reinforcement learning according to claim 1, characterized in that, The updated local policy parameters are obtained specifically through: Obtain a set of reinforcement learning interaction samples associated with local node identifiers, and construct an online policy update batch based on the set of reinforcement learning interaction samples; The strategy improvement target value is calculated based on the online strategy update batch. The strategy improvement target value is the weighted combination of the time cumulative weighted result of the reward value sequence and the estimated return value corresponding to the joint state vector sequence at the next time step. In the process of calculating the target value of strategy improvement, collaborative constraint information is introduced. The collaborative constraint information is converted into constraint penalty value and combined with the target value of strategy improvement in a weighted manner to obtain the constraint update target value. The constraint penalty value is the weighted sum of the target global indicator constraint penalty value, resource competition constraint penalty value, strategy consistency constraint penalty value and security boundary constraint penalty value. The policy parameter update amount is calculated based on the constraint update target value. The policy parameter update amount is the product of the gradient direction of the constraint update target value with respect to the local policy parameter and the preset learning rate. The updated local policy parameters are obtained by applying the policy parameter update amount to the local policy parameters, and then the updated local policy parameters are written into the policy parameter storage structure.

8. The distributed collaborative management and control method based on reinforcement learning according to claim 1, characterized in that, The execution of the command specifically includes: Obtain the current joint state vector and read the updated local policy parameters. Based on the updated local policy parameters, perform action selection calculation on the current joint state vector to obtain the target action. Perform constraint verification on the target action to obtain the constraint verification result; When the constraint verification result indicates that the amplitude constraint and frequency constraint are satisfied, the target action is parsed into the target running parameter identifier, the target parameter value and the effective time identifier according to the mapping relationship between the parameter adjustment instruction and the parameter read and write channel, and a parameter write message is formed based on the parameter read and write address; Based on the parameter write message, the target action is sent to the corresponding controlled device through the parameter read and write channel. On the controlled device side, the target parameter value is written to the target operating parameter according to the target operating parameter identifier. At the effective time corresponding to the effective time identifier, the operating parameter set update operation is performed and the operating parameter adjustment process is triggered.

9. The distributed collaborative management and control method based on reinforcement learning according to claim 1, characterized in that, The generation of the dynamic collaborative optimization results and the strategy consistency control results specifically includes: After the target action is sent through the parameter read / write channel, the performance feedback information of the corresponding controlled device is collected, and a feedback summary is generated based on the performance feedback information. The communication neighbor feedback summary is received through the communication link, and a consistency deviation metric is calculated based on the local feedback summary and the communication neighbor feedback summary. The consistency deviation metric is the weighted sum of the deviations between the local feedback summary and the communication neighbor feedback summary on key feedback indicators. The consistency deviation metric is compared with the consistency threshold. When the consistency deviation metric exceeds the consistency threshold, the Gossip protocol consistency propagation process is initiated. When the consistency deviation metric does not exceed the consistency threshold, the feedback digest collection and consistency deviation metric calculation are maintained. In the Gossip protocol consensus propagation process, a fixed number of propagation operations are performed sequentially according to the communication cycle. In each propagation operation, a communication neighbor is determined based on the adjacency topology, feedback digests are exchanged with the communication neighbor, and a weighted aggregation operation is performed on the exchanged feedback digests to update the consensus correction digest. After completing the fixed number of propagation operations, the consensus correction digest is output. Perform consistency correction updates on local policy parameters based on the consistency correction summary; The dynamic collaborative optimization result is obtained by calculating the policy output of the current joint state vector based on the corrected local policy parameters. The convergence degree of the consistency deviation is calculated based on the consistency deviation metric and the policy parameter correction amount. The policy consistency control result is generated by hierarchical judgment based on the convergence degree of the consistency deviation.

Images