A strategy optimization method for cooperative control of multi-agent system

Abstract

The application relates to the technical field of multi-agent system cooperative control, and discloses a strategy optimization method for multi-agent system cooperative control, which comprises the following steps: collecting local observation data and converting the same to obtain a state representation vector; receiving a feature flow of an adjacent agent to project an output intention feature vector; combining the two to complete mutual information lower bound maximization calculation and output a credit distribution weight; calculating an advantage function and correcting a strategy gradient to update strategy network parameters; and rewriting a configuration register according to a dispersion factor to adjust a sliding window step number and a quantization bit width. The application decouples uncertainty of a transition probability through mutual constraint of local states and potential intentions, reduces gradient variance fluctuation, reduces communication overhead while avoiding control deadlock, and enhances convergence stability of a decentralized network.

Description

Technical Field

[0001] This invention relates to a strategy optimization method for cooperative control of multi-agent systems, belonging to the field of cooperative control technology for multi-agent systems. Background Technology

[0002] Autonomous decision-making methods based on decentralized reinforcement learning have been widely applied to multi-agent cooperative control. Each control node collects local observation data through sensors, and then uses a local policy network to calculate the control output and drive the actuator, thereby completing group cooperation without central scheduling. Under the decentralized Markov decision framework, each node gradually establishes the correspondence between environmental state and control action by continuously iterating the policy gradient, enabling the system to have autonomous adaptation and distributed cooperative capabilities.

[0003] In the process of multiple nodes updating policies simultaneously, the policy parameters of each node are constantly changing. For any node, even if it takes the same action in the same local state, the state transition probability and reward feedback of the environment may change due to the policy changes of neighboring nodes. Therefore, the training environment exhibits obvious non-stationarity. Traditional credit allocation methods usually assume that the global state can be obtained in real time or that the communication bandwidth is unlimited. Under decentralized or communication-limited conditions, nodes lack global reference information and find it difficult to accurately distinguish the marginal contribution of their own actions to the global goal, which easily leads to inaccurate credit allocation and increases the variance of policy gradient estimation. Specifically, existing technologies mainly have the following problems: First, they rely too much on the transmission of full state information. For example, Chinese invention patent CN117173268B discloses a multi-agent reinforcement learning collaborative method based on dynamic graph communication, which alleviates the problem of... While communication introduces topology rigidity issues, in complex scenarios with high dynamics and strong interference, the dynamic graph update mechanism it relies on increases the computational overhead of computing nodes in graph maintenance. Furthermore, due to the over-reliance on the strong correlation and coupling of neighboring node action features, the policy network oscillates when data link jitter occurs, reducing the convergence speed of action policies and making it difficult to achieve smooth convergence of action policies within extremely short time slots. To address these issues, existing technologies typically increase the communication bandwidth between nodes or set up centralized monitoring servers. However, as the number of nodes increases, communication overhead and hardware costs increase. High-frequency transmission of full information also increases communication latency, affecting the timeliness of control response. Another approach is to ignore the changes in neighboring node policies and treat the local environment as a static Markov process. However, this can cause different states to be confused in feature representations and lead to control failure of the policy network when subjected to dynamic interference.

[0004] Therefore, under decentralized and limited communication bandwidth conditions, how to distinguish the impact of simultaneous policy updates by multiple nodes on the probability of environmental state transition, achieve accurate credit allocation, and ensure stable convergence of policy gradients has become a technical problem that multi-agent cooperative control methods need to solve. Summary of the Invention

[0005] To address the problems in the background art, the technical solution of the present invention is as follows: A strategy optimization method for cooperative control of a multi-agent system, comprising the following steps:

[0006] Step S101: Obtain the local raw observation data vector collected by the environmental state acquisition unit, input it into the temporal feature extraction network to complete the temporal context association calculation, and output the local state representation vector;

[0007] Step S102: Obtain the action indicator feature stream received by the wireless communication interface, input it into the intent coding network to complete the nonlinear feature projection, and output the potential intent feature vector.

[0008] Step S103: Input the local state representation vector and the latent intention feature vector into the mutual information estimation model to complete the calculation of the lower bound of mutual information and output the local credit allocation weight.

[0009] Step S104: Calculate the advantage function based on the local task reward input from the environment, multiply correct the local policy gradient output by the local policy network using local credit allocation weights to generate the local policy update gradient, update the internal connection weights of the local policy network based on the local policy update gradient, and output the multidimensional control instruction vector to the state transition unit.

[0010] A reverse adjustment loop is constructed between steps S101 and S104: the computing node reads the norm of the local policy update gradient in the last 10 cycles from memory, calculates the variance of the norm and multiplies it with the Shannon information entropy of the current action probability distribution vector of the local policy network to obtain the credit distribution dispersion factor. When the credit distribution dispersion factor is greater than 0.85, the configuration register is automatically rewritten, the sliding window history step number parameter of the temporal feature extraction network is adjusted from 3 to 8, and the received quantization bit width of the action indicator feature stream is switched from 4 bits to 16 bits.

[0011] Preferably, the calculation of the credit distribution dispersion factor includes the following steps: opening a circular buffer of fixed length 10 in the memory of the computing node, cyclically reading and storing the norm of the local policy update gradient in the most recent 10 periods, and calculating the sample variance of the norm change in the circular buffer; calculating the Shannon information entropy of the current action probability distribution vector of the local policy network output layer; and using the on-chip floating-point computing unit to multiply the sample variance and the Shannon information entropy to output the credit distribution dispersion factor.

[0012] Preferably, in step S101, the temporal feature extraction network adopts a two-layer bidirectional gated recurrent unit network structure with 64 hidden layer neurons; each computing node reads environmental obstacle distribution data in a loop through the synchronous serial peripheral interface and external memory interface, and combines it with the spatial motion acceleration data of the computing node collected by the inertial navigation unit, and forms a local original observation data vector by splicing and combining the environmental obstacle distribution data and the spatial motion acceleration data.

[0013] Preferably, in step S102, each computing node receives the action indicator feature stream through the data transmission chip via the universal asynchronous transceiver interface; the action indicator feature stream is a 16-bit binary data message, and the action indicator feature stream is transformed and output as a 128-dimensional potential intent feature vector by the on-chip floating-point matrix multiplier in the intent coding network to complete the nonlinear feature projection mapping.

[0014] Preferably, in step S103, the mutual information estimation model constructs a lower bound loss function for mutual information based on a contrastive learning mechanism, updates the correlation mapping weight matrix through stochastic gradient descent to maximize the correlation between positive sample pairs of local state representation vectors and potential intention feature vectors, while minimizing the correlation with negative samples of non-matching intention features randomly sampled from the experience replay area, and calculates the output local credit assignment weight.

[0015] Preferably, in step S104, limiting the boundary self-constraint range of the local policy update gradient includes the following sub-steps: Step S1041, calculate the cosine similarity between the local policy update gradient and the historical average update gradient in the last 5 periods; Step S1042, when the cosine similarity is lower than the safe similarity threshold of 0.30 and the calculated local credit allocation weight exceeds the preset range, call the projection boundary constraint operator to limit the local credit allocation weight to between 0.10 and 0.30, and use the Lagrange multiplier method to project the current local policy update gradient vector back into the safe manifold space of the convex hull centered on the historical average update gradient to obtain the constrained update gradient.

[0016] Preferably, in step S104, the advantage function is calculated using a generalized advantage estimation structure based on local reward and discount factor; the multi-dimensional control command vector obtained by the computing node includes multi-axis motion setpoints for the distributed controlled object.

[0017] Preferably, in step S104, the multidimensional control command vector is converted by the microprocessor's timer to output a pulse width modulation signal with a variable duty cycle. The pulse width modulation signal is output to the state transformation unit through the microprocessor's physical pins to adjust the instantaneous speed of the motor at a frequency of 50Hz, thereby changing the spatial displacement state of the distributed controlled object to complete the adaptive cancellation of time-varying horizontal physical disturbances.

[0018] Preferably, the reverse adjustment loop further includes the following steps: when the credit distribution dispersion factor is continuously lower than the dispersion lower limit threshold of 0.20, the computing node rewrites the configuration register, dynamically adjusts the sliding window historical step number parameter in the time series feature extraction network from 3 to 2, and compresses the received quantization bit width of the action indicator feature stream from 16 bits to 4 bits.

[0019] Compared with the prior art, the beneficial effects of the present invention are:

[0020] 1. In the policy optimization of cooperative control of multi-agent systems, by forming local state representation vectors from local observation data and converting the action indicator feature streams sent by neighboring nodes into potential intention feature vectors, and then determining the local credit allocation weights based on the degree of correlation between the two, each node can still accurately judge the contribution of its own actions to the cooperative task even when it cannot obtain the global state. This can reduce the impact of environmental non-stationarity caused by simultaneous policy updates of multiple nodes on credit allocation, reduce the variance of local policy gradient estimation, and enable the control policy to maintain relatively stable updates and convergence under the condition of small communication data volume.

[0021] 2. By dynamically adjusting the sliding window historical steps and the received quantization bit width of the action indicator feature stream of the temporal feature extraction network according to the credit distribution dispersion factor, the feature extraction and communication accuracy can change with the policy update state. When the policy update fluctuates greatly, longer historical observation information and higher reception accuracy can be used to reduce the impact of feature noise on the training process. When the policy tends to be stable, the amount of historical data that needs to be processed can be reduced and the amount of communication data can be reduced, thus balancing the stability of policy update, computational overhead and channel occupancy.

[0022] 3. By judging the continuous loss of communication data packets and using the potential intent feature vector of the previous normal time slot for attenuation prediction when packet loss occurs, the impact of short-term communication interruption on the current decision can be reduced. When the continuous packet loss exceeds the preset fault threshold, the system enters the degradation mode and no longer uses the missing adjacency information to calculate the local credit allocation weight, thereby avoiding the wrong policy update direction due to incomplete adjacency information. Even if affected by strong wireless interference or continuous packet loss, each node can still maintain a stable control response and smoothly transition to the independent update state. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the optimization of the multi-agent cooperative control strategy of the present invention;

[0024] Figure 2 This is a state transition diagram for the cooperative control of the multi-agent system of the present invention.

[0025] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0027] A strategy optimization method for cooperative control of a multi-agent system includes the following steps:

[0028] Step S101: Obtain the local raw observation data vector collected by the environmental state acquisition unit, input it into the temporal feature extraction network to complete the temporal context association calculation, and output the local state representation vector;

[0029] Step S102: Obtain the action indicator feature stream received by the wireless communication interface, input it into the intent coding network to complete the nonlinear feature projection, and output the potential intent feature vector.

[0030] Step S103: Input the local state representation vector and the latent intention feature vector into the mutual information estimation model to complete the calculation of the lower bound of mutual information and output the local credit allocation weight.

[0031] Step S104: Calculate the advantage function based on the local task reward input from the environment, multiply correct the local policy gradient output by the local policy network using local credit allocation weights to generate the local policy update gradient, update the internal connection weights of the local policy network based on the local policy update gradient, and output the multidimensional control instruction vector to the state transition unit.

[0032] A reverse adjustment loop is constructed between steps S101 and S104: the computing node reads the norm of the local policy update gradient in the last 10 cycles from memory, calculates the variance of the norm and multiplies it with the Shannon information entropy of the current action probability distribution vector of the local policy network to obtain the credit distribution dispersion factor. When the credit distribution dispersion factor is greater than 0.85, the configuration register is automatically rewritten, the sliding window history step number parameter of the temporal feature extraction network is adjusted from 3 to 8, and the received quantization bit width of the action indicator feature stream is switched from 4 bits to 16 bits.

[0033] Preferably, the calculation of the credit distribution dispersion factor includes the following steps: opening a circular buffer of fixed length 10 in the memory of the computing node, cyclically reading and storing the norm of the local policy update gradient in the most recent 10 periods, and calculating the sample variance of the norm change in the circular buffer; calculating the Shannon information entropy of the current action probability distribution vector of the local policy network output layer; and using the on-chip floating-point computing unit to multiply the sample variance and the Shannon information entropy to output the credit distribution dispersion factor.

[0034] Preferably, in step S101, the temporal feature extraction network adopts a two-layer bidirectional gated recurrent unit network structure with 64 hidden layer neurons; each computing node reads environmental obstacle distribution data in a loop through the synchronous serial peripheral interface and external memory interface, and combines it with the spatial motion acceleration data of the computing node collected by the inertial navigation unit, and forms a local original observation data vector by splicing and combining the environmental obstacle distribution data and the spatial motion acceleration data.

[0035] Preferably, in step S102, each computing node receives the action indicator feature stream through the data transmission chip via the universal asynchronous transceiver interface; the action indicator feature stream is a 16-bit binary data message, and the action indicator feature stream is transformed and output as a 128-dimensional potential intent feature vector by the on-chip floating-point matrix multiplier in the intent coding network to complete the nonlinear feature projection mapping.

[0036] Preferably, in step S103, the mutual information estimation model constructs a lower bound loss function for mutual information based on a contrastive learning mechanism, updates the correlation mapping weight matrix through stochastic gradient descent to maximize the correlation between positive sample pairs of local state representation vectors and potential intention feature vectors, while minimizing the correlation with negative samples of non-matching intention features randomly sampled from the experience replay area, and calculates the output local credit assignment weight.

[0037] Preferably, in step S104, limiting the boundary self-constraint range of the local policy update gradient includes the following sub-steps: Step S1041, calculate the cosine similarity between the local policy update gradient and the historical average update gradient in the last 5 periods; Step S1042, when the cosine similarity is lower than the safe similarity threshold of 0.30 and the calculated local credit allocation weight exceeds the preset range, call the projection boundary constraint operator to limit the local credit allocation weight to between 0.10 and 0.30, and use the Lagrange multiplier method to project the current local policy update gradient vector back into the safe manifold space of the convex hull centered on the historical average update gradient to obtain the constrained update gradient.

[0038] Preferably, in step S104, the advantage function is calculated using a generalized advantage estimation structure based on local reward and discount factor; the multi-dimensional control command vector obtained by the computing node includes multi-axis motion setpoints for the distributed controlled object.

[0039] Preferably, in step S104, the multidimensional control command vector is converted by the microprocessor's timer to output a pulse width modulation signal with a variable duty cycle. The pulse width modulation signal is output to the state transformation unit through the microprocessor's physical pins to adjust the instantaneous speed of the motor at a frequency of 50Hz, thereby changing the spatial displacement state of the distributed controlled object to complete the adaptive cancellation of time-varying horizontal physical disturbances.

[0040] Preferably, the reverse adjustment loop further includes the following steps: when the credit distribution dispersion factor is continuously lower than the dispersion lower limit threshold of 0.20, the computing node rewrites the configuration register, dynamically adjusts the sliding window historical step number parameter in the time series feature extraction network from 3 to 2, and compresses the received quantization bit width of the action indicator feature stream from 16 bits to 4 bits.

[0041] Example 1: Each Automated Guided Vehicle (AGV) is equipped with a 400MHz microprocessor, a 2MB static random access memory, and a 900MHz wireless communication module. Using the microprocessor as the computing node, the AGVs work together to perform material transport tasks within a 100m long and 50m wide bidirectional narrow-channel grid storage area. Due to the spatial constraints of the grid intersections and the overlapping of vehicle trajectories, the AGVs may experience local deadlocks and stalls due to mutual avoidance even without physical collisions. The average running speed drops below 0.1m / s within 60 seconds.

[0042] At mission start, each automated guided vehicle discretizes its travel path into a state array represented by two-dimensional grid coordinates and pre-configures a local policy network weight table represented by a two-dimensional floating-point matrix in static random access memory. During operation, the environmental state acquisition unit reads the pulse count of the wheel encoder and the motor drive current every 100ms. The instantaneous speed of the vehicle is calculated from the pulse count, and the current grid position coordinates, current speed, and motor drive current are written into a state transition array of length 1000 in the sampling order to record vehicle state changes and determine implicit deadlock events.

[0043] Within the same sampling period, the computing node reads environmental obstacle distribution data from the external memory cyclically through the synchronous serial peripheral interface and obtains spatial motion acceleration data of the computing node collected by the inertial navigation unit. The environmental obstacle distribution data and the spatial motion acceleration data of the computing node are concatenated to form a local raw observation data vector. The local raw observation data vector is input into the temporal feature extraction network. The temporal feature extraction network adopts a two-layer bidirectional gated recurrent unit network structure with 64 hidden layer neurons. It performs temporal context association calculation on the input data within the continuous sampling period and outputs a local state representation vector.

[0044] Adjacent automated guided vehicles encode the avoidance control commands currently output by the local policy network into a 16-bit binary action indicator feature stream and send it to adjacent vehicles via a wireless communication module. Each computing node receives the action indicator feature stream through the universal asynchronous transceiver interface of the data transmission chip and inputs it into the intent encoding network. The intent encoding network uses an on-chip floating-point matrix multiplier to perform matrix operations and nonlinear feature projection, outputting a 128-dimensional latent intent feature vector. The mutual information estimation model takes the local state representation vector and the latent intent feature vector within the same sampling period as positive sample pairs, and randomly extracts non-matching intent feature negative samples from other sampling periods from the experience replay area. The computing nodes calculate the correlation degree of the positive sample pairs and the negative sample pairs respectively, update the correlation mapping weight matrix through stochastic gradient descent, and normalize the correlation degree of the positive sample pairs to obtain the local credit allocation weight.

[0045] When the motor drive current is greater than 2A for 30 consecutive sampling points, and the instantaneous driving speed calculated from the wheel encoder pulse count is less than 0.05m / s for 30 consecutive sampling points, the computing node determines that the current state is an implicit deadlock event caused by path conflict. Subsequently, the current grid position coordinates are associated with the avoidance control command generated in the previous sampling period. An implicit negative event state transition tuple with an additional negative feedback weight coefficient of -5 is generated in the static random access memory. The tuple is written to the head of the experience playback cache array consisting of 200 storage units. The implicit negative event state transition tuple is a five-dimensional data structure, which sequentially contains the current grid position coordinates, the avoidance control command, the local task report input by the environment, the next grid position coordinates, and the negative feedback weight coefficient of -5.

[0046] During each training cycle, the microprocessor reads implicit negative event state transition tuples from the experience replay cache array, adds the negative feedback weight coefficient -5 to the local task reward in the tuple, and increases the absolute value of the one-step time difference error corresponding to the avoidance control command that causes implicit deadlock. The computing node discounts and accumulates the one-step time difference error of the continuous sampling cycle according to the local task reward and discount factor through the generalized advantage estimation structure to obtain the advantage function, and outputs the local policy gradient by the local policy network. Subsequently, the local policy gradient is multiplicatively corrected by the local credit allocation weight to generate the local policy update gradient, and then the internal connection weights of the local policy network are updated by stochastic gradient descent. After the update, the local policy network reduces the output probability of the avoidance control command that causes implicit deadlock, so that the AGV adjusts its travel speed 1 second in advance when it runs to the same grid position again.

[0047] Each computing node completes local raw observation data vector processing, implicit negative event recording, local credit allocation weight calculation, and local policy network update at the vehicle control terminal. Basic heartbeat information and action indicator feature streams are transmitted between adjacent vehicles. The wireless communication module does not transmit global trajectory data. In a continuous 10-hour material transportation test, the deadlock stagnation time caused by mutual avoidance of each automated guided vehicle was reduced from an average of 320 seconds per hour to an average of 12 seconds per hour. The channel occupancy rate of the wireless communication module was reduced by 85%. In the distributed control process, each automated guided vehicle identifies implicit deadlock events based on locally collected data, writes the corresponding negative feedback samples into the experience replay cache array, and uses the action indicator feature streams sent by adjacent vehicles to calculate local credit allocation weights, so that the local policy network of each vehicle is continuously updated under the condition of limited communication bandwidth.

[0048] Example 2: Deployment of the Computational Chip for the Multi-Autonomous Guiding Workstation Carrier Cooperative Controller. This example's strategy optimization method controls multiple autonomous guiding workstation carriers to run along a preset industrial logistics channel. Each carrier's environmental status acquisition unit collects its current position coordinates and current speed. The computing chip converts the collected results into a local raw observation data vector. ,in, As a time index, the local original observation data vector satisfies: ,in, and These represent the vehicle's current position along the X and Y axes in the global coordinate system, respectively. and The current velocities of the vehicle along the X and Y axes in the global coordinate system are respectively. The computing chip inputs the continuously obtained local raw observation data vectors into the temporal feature extraction network in the sampling order to complete the temporal context association calculation and output the local state representation vector. The local state representation vector and the potential intention feature vector of the adjacent vehicles are used together to calculate the local credit allocation weight.

[0049] The computing chip simultaneously converts the local raw observation data vector The input is a local policy network, which calculates a multi-dimensional control command vector based on the current position and current velocity. It is represented as: ,in, The target acceleration is defined as -2.5 m / s² to 2.5 m / s². The target turning angle is defined as -0.52 rad to 0.52 rad. The local policy network consists of an input layer, a hidden layer, and an output layer. The input layer has 4 neurons, the hidden layer adopts a two-layer fully connected network structure with 64 hidden neurons in each layer, and the output layer has 2 neurons.

[0050] Adjacent vehicles exchange current multidimensional control command vectors via wireless communication modules. The transmitting end encodes the multidimensional control command vectors into a 16-bit binary action indicator feature stream. The receiving end receives the action indicator feature stream through the universal asynchronous transceiver interface of the data transmission chip and inputs it into the intent encoding network to complete nonlinear feature projection, obtaining a 128-dimensional latent intent feature vector. The mutual information estimation model outputs local credit allocation weights based on the correlation between the local state representation vector and the latent intent feature vector. The computing chip also calculates the cooperative deviation based on the current position of adjacent vehicles and their multidimensional control command vectors. The collaborative deviation is included in the local task reward based on environmental input. .

[0051] Local task reward satisfaction: ,in, The target state vector is preset. This is the position error weighting coefficient, and its value is set to 0.6. The coordination deviation weighting coefficient is set to 0.4 for the numbered... For vehicles whose cooperative deviation satisfies: ,in, In order to be with the first A set of adjacent vehicles, For adjacent vehicles, For the first The vehicle and the first The weight of the communication link between each vehicle is set to 0.5; For the first Each vehicle in time Multidimensional control command vector at time. For the first Each vehicle in time Multidimensional control command vector at time.

[0052] Before calculating the local task reward, the computing chip performs dimensionless processing on the physical quantities in the local raw observation data vector and the multidimensional control command vector. For the position component, the current horizontal and vertical coordinates are divided by the length of the storage and logistics channel (100m) and the width (50m), respectively, to obtain the position ratio between 0 and 1. For the velocity component, the current speed is divided by the maximum allowable speed of the vehicle (2.5m / s). For the control component, the target acceleration is divided by the upper limit of acceleration (2.5m / s²), and the target steering angle is divided by the upper limit of steering angle (0.52rad). After processing, each component is within the dimensionless numerical range of -1 to 1. The computing chip calls the floating-point computing unit to perform the L2 norm square operation and sums the position error term and the cooperative deviation according to the corresponding weights, so that the two participate in the local task reward calculation with a unified numerical scale.

[0053] During training, the computing chip calculates the advantage function based on the local task reward, obtains the local policy gradient from the local policy network, and then uses the local credit allocation weight to multiplicatively correct the local policy gradient to generate the local policy update gradient. Based on this, the internal connection weights of the local policy network are updated. When the cooperative bias increases, the corresponding deduction in the local task reward increases accordingly. The parameter update thereby reduces the output probability of the multidimensional control command vector with large differences in actions between adjacent vehicles.

[0054] During the control output phase, the microprocessor's timer converts the multidimensional control command vector into a pulse width modulation signal with a variable duty cycle, and outputs it to the state transition unit through a physical pin at a frequency of 50Hz. The state transition unit adjusts the current of the drive motor and the deflection angle of the steering motor according to the pulse width modulation signal, and outputs the target acceleration and the target steering angle respectively to change the spatial displacement state of the vehicle.

[0055] Test group 1, test group 2, control group 1 and control group 2 were set up on the actual material conveying track. Test group 1 and test group 2 adopted the strategy optimization method of this embodiment. Control group 1 and control group 2 served as corresponding references. During the operation of each group, the position, speed and energy consumption data of the vehicle were continuously recorded. The average value of collaborative tracking error was calculated based on the distance between the actual trajectory of the vehicle and the target trajectory. The average energy consumption was calculated based on the power data during the operation. The frequency of collision risk was also counted.

[0056] After the aforementioned state acquisition, strategy calculation, local credit allocation, and gradient update, Test Group 1 achieved a mean collaborative tracking error of 0.034m, a mean energy consumption of 112.4W, and zero collision risks. Test Group 2, using the same processing procedure, achieved a mean collaborative tracking error of 0.038m, a mean energy consumption of 115.8W, and zero collision risks. As a reference, Control Group 1 achieved a mean collaborative tracking error of 0.124m, a mean energy consumption of 142.1W, and two collision risks; Control Group 2 achieved a mean collaborative tracking error of 0.089m, a mean energy consumption of 128.5W, and one collision risk. As can be seen from the above records, the mean collaborative tracking error of both test groups is below 0.04m, and the mean energy consumption is below 116W.

[0057] Example 3: This example deploys the strategy optimization method in a system containing... In a multi-agent system with n agents, each agent updates its local policy network parameters through local state interaction and distributed computation. For the nth agent... An intelligent agent. At any given moment Collect local raw observation data vectors , by parameters Local policy network according to policy Generate Actions ,action As a multidimensional control command vector output by the agent, policy parameters The update process uses a local evaluation function. Evaluate the outcome of actions in the current state.

[0058] Within each training time slot, the temporal feature extraction network performs temporal context association calculation based on continuously collected local raw observation data vectors, and outputs the local state representation vector of the current agent. Neighboring agents encode the actions output by the local policy network into action indicator feature streams and send them through the wireless communication interface. After receiving the action indicator feature stream, the current agent inputs it into the intent encoding network for nonlinear feature projection to obtain the latent intent feature vector.

[0059] When determining the local credit allocation weights, the processor forms positive sample pairs by combining the local state representation vector and the latent intention feature vector corresponding to the same training time slot and the same agent. It also extracts non-matching intention feature vectors generated from other historical time slots in the experience replay area as negative samples through random indexing. The on-chip floating-point computing unit first calculates the dot product of the two vectors in the positive sample pair, and then calculates the dot product of the local state representation vector and multiple negative sample vectors. The dot product results are then non-linearly scaled using an exponential function. Subsequently, the ratio of the positive sample exponent value to the sum of all positive and negative sample exponent values ​​is calculated, and the logarithm of the ratio is taken to obtain the lower bound loss value of the mutual information of the current sample pair. The processor uses stochastic gradient descent to reduce the loss value and iteratively updates the correlation mapping weight matrix to increase the correlation between matching vectors and decrease the correlation between non-matching vectors. Finally, the correlation of the positive sample pairs is normalized, and the local credit allocation weights are output.

[0060] Each agent also obtains information about neighboring agents through communication links. Local raw observation data vector ,by Represents intelligent agents With intelligent agents The weight of the communication link between them Represents intelligent agents The neighborhood group, Represents intelligent agents The confidence weights for the original local observation data vector satisfy the following conditions: intelligent agent Construct the collaborative observation vector according to the following formula. : Local policy networks based on cooperative observation vectors The basic expression for calculating the unweighted local policy gradient is: ,in, This represents a policy optimization function that aims to accumulate local task rewards. Represents intelligent agents At any moment Local task rewards derived from environmental input; This is a discount factor with a value of 0.95. Represents intelligent agents The value function, The expression represents the expected operation on the training samples. The calculation result in parentheses is used as the time difference residual of the current period. The processor discounts and accumulates the time difference residuals of the current period and subsequent periods according to the generalized advantage estimation structure to obtain the advantage function. Based on this, the local policy gradient is calculated. After the mutual information estimation model outputs the local credit allocation weights, the processor uses the weights to multiplicatively correct the local policy gradient to generate the local policy update gradient. Then, the internal connection weights of the local policy network are updated according to the local policy update gradient.

[0061] After completing the local policy update gradient calculation, the processor reads the local policy update gradient of the current cycle and the historical update gradients of the last 5 cycles from the on-chip memory, calculates the historical average update gradient, and obtains the cosine similarity by dividing the dot product of the current gradient and the historical average update gradient by the product of their magnitudes. When the cosine similarity is lower than the safety similarity threshold of 0.30, and the current local credit allocation weight exceeds 0.10 to 0.30, the processor starts the projection boundary constraint operator. This operator, as a data boundary truncation and safety limiting program executed in the microprocessor's on-chip ROM, takes the current local credit allocation weight, gradient update vector, and boundary constants of 0.10 and 0.30 as input variables. By performing a one-dimensional numerical saturation limiting operation in the floating-point processor register, when the local credit allocation weight is lower than 0.10, the value is forcibly assigned to 0.10; when the weight is higher than 0.30, the value is forcibly assigned to 0.30. The closed-loop output of the weight factor after truncation constraint is used to first limit the local credit allocation weight to the range of 0.10 to 0.30.

[0062] During gradient projection, the processor constructs a constraint point set using the historical update gradients of the last 5 cycles and their historical average update gradients. It then takes a convex combination of gradient vectors from the constraint point set and, with the historical average update gradient as the center and a preset maximum allowable error as the boundary, constructs a closed convex cellular polyhedron enclosed by the constraint point set in the computational high-dimensional real vector space. This convex cellular polyhedron, as a flat local Euclidean subspace manifold with zero curvature, has its geometric piecewise linear boundary uniquely determined by the convex combination of the historical update gradient vectors of the last 5 cycles. This eliminates the geometric conflict between the linear envelope boundary and the nonlinear manifold under multi-directional temporal gradient flow, thus forming a closed convex hull safe manifold space. Subsequently, the current local policy update gradient is used as the vector to be projected. With the objective of minimizing the distance between the two gradient vectors before and after projection, the Lagrange multiplier method is called to solve the convex quadratic programming problem, obtaining the projection vector located within the convex hull safe manifold space. The original local policy update gradient is then replaced by the projection vector.

[0063] To handle situations where the communication link is affected by environmental noise interference, the system sets a communication stability threshold. ,by Represents intelligent agents With intelligent agents The link signal strength between them, when At that time, intelligent agent intelligent agents Corresponding communication link weight Reset to 0 and adjust its own confidence weight. This ensures that the weights still satisfy the normalization relation, when When set to 0.62, under communication interference conditions, the average cumulative reward of the agent swarm during training... After several iterations, the system tends to stabilize, with fluctuations remaining within 2%.

[0064] The upper threshold of the credit distribution dispersion factor (0.85) and the lower threshold of the lower threshold (0.20) are determined based on offline Monte Carlo simulation results and industrial field measurement data. The factor is obtained by multiplying the variance of the local policy update gradient norm over the most recent 10 periods by the Shannon information entropy of the current action probability distribution vector of the local policy network. To eliminate the dimensional breakage and decision boundary drift caused by large changes in the absolute value of the gradient, the microprocessor first divides the variance of the gradient norm sample over the most recent 10 sampling periods by the square of the mean of the gradient norm over these 10 periods when calculating the factor, converting it into a dimensionless gradient dispersion coefficient value. Then, it divides the current Shannon information entropy value by the natural logarithm of the dimension of the current output action probability distribution vector of the local policy network, converting it into a dimensionless probability confusion factor. The gradient dispersion coefficient is multiplied by the probability confusion ratio to standardize the credit distribution dispersion factor to a dimensionless value between 0 and 1.0. When the credit distribution dispersion factor is greater than 0.85, the gradient fluctuation and action selection uncertainty in the last 10 periods are at a high level. The computing node rewrites the configuration register, adjusts the sliding window history step parameter of the time series feature extraction network from 3 to 8, and switches the received quantization bit width of the action indicator feature stream from 4 bits to 16 bits. When the credit distribution dispersion factor calculated during a complete 10-period ring buffer update is less than 0.20, the computing node adjusts the sliding window history step parameter from 3 to 2 and compresses the received quantization bit width from 16 bits to 4 bits.

[0065] The safety similarity threshold of 0.30 corresponds to an angle of approximately 72.5 degrees between two gradient vectors. When the cosine similarity between the current local policy update gradient and the historical average update gradient is less than 0.30, their directional deviation exceeds 72.5 degrees, triggering gradient projection processing. The local credit allocation weight is limited to a range of 0.10 to 0.30. When the weight is less than 0.10, the proportion of the current agent's local contribution in the gradient update decreases. When the weight is greater than 0.30, the gradient change of the current agent has a greater impact on the parameter updates of neighboring agents. Therefore, it is limited to the above range.

[0066] Strategy parameters A synchronous update is performed every 100 time steps. The computing nodes utilize existing processor resources to complete local gradient aggregation and parameter writing. The temporal feature extraction network adopts a two-layer bidirectional gated recurrent unit network structure with 64 hidden layer neurons. It receives the input state of 8 time steps according to a fixed maximum input dimension. When the sliding window history step number parameter is adjusted from 3 to 8, the processor sequentially packages the local raw observation data vectors of the most recent 8 time steps and inputs them into the network. When the parameter is adjusted from 3 to 2, only the data of the latest 2 time steps are extracted, and zero values ​​are filled in the input positions of the remaining 6 time steps to keep the network input dimension unchanged.

[0067] When the received quantization bit width of the action indicator feature stream switches between 4 bits and 16 bits, upon receiving the switching configuration command, the transmitting end first inserts a 2-byte feature frame synchronization check sequence into the header of the action indicator feature stream data packet. For a 16-bit received quantization bit width, hexadecimal code EB90 is inserted, and for a 4-bit received quantization bit width, hexadecimal code 5A is inserted. The receiving end matches this feature frame synchronization check sequence through the hardware timing detector of the on-chip universal asynchronous transceiver to realign the data frame boundaries and complete timing synchronization, and synchronously rewrites the transmission length register of the direct memory access controller. The data receiving end unifies the input value range through the dynamic scaling alignment module. When the received quantization bit width is compressed from 16 bits to 4 bits, the processor receives 4 bits of fixed-point binary data, shifts the data 12 bits to the left to expand it into a 16-bit unsigned fixed-point number, and pads it with zeros at the low bits. Subsequently, it is multiplied by a normalization conversion ratio by a multiplier to map it to the same dynamic value range as the 16-bit floating-point data, and then input into the on-chip floating-point matrix multiplier in the intent coding network.

[0068] The system continuously monitors the output actions of each agent. The deviation from the target trajectory is calculated, and the control gain parameters of each agent are adjusted accordingly. When the local deviation continues to increase, the agent... Reduce the corresponding communication link weight At the same time, adjust its own confidence weight. To maintain weight uniformity, the proportion of neighboring agent state information in the current decision is reduced accordingly.

[0069] Example 4: This example combines Figures 1 to 2 This paper describes a strategy optimization method for cooperative control of a multi-agent system, such as... Figure 1 As shown, in step S101, the local raw observation data vector collected by the environmental state acquisition unit is obtained and input into the temporal feature extraction network to complete the temporal context association calculation and output the local state representation vector. Then, in step S102, the action indicator feature stream received by the wireless communication interface is obtained and input into the intention encoding network to complete the nonlinear feature projection and output the potential intention feature vector. Then, in step S103, the local state representation vector and the potential intention feature vector are input into the mutual information estimation model to complete the mutual information lower bound maximization calculation and output the local credit allocation weight. Finally, in step S104, the advantage function is calculated based on the local task reward input by the environment, and the local policy gradient output by the local policy network is multiplicatively corrected using the local credit allocation weight to generate the local policy update gradient. The internal connection weights of the local policy network are updated according to the local policy update gradient, and the multidimensional control command vector is output to the state transformation unit.

[0070] like Figure 2 As shown, if the multi-agent system cooperative control encounters path conflicts during execution, an implicit deadlock event will occur. After determining the implicit deadlock event, the system adjusts the travel speed in advance to provide feedback and regulate the multi-agent system cooperative control. When the continuous packet loss of the communication link exceeds the preset fault threshold, the system switches to a degraded mode and smoothly transitions to an independent update state. When the multi-agent system cooperative control achieves the maintenance of the cooperative task rate, it enters a steady-state cooperative control interval. Within the steady-state cooperative control interval, the multi-agent system cooperative control is readjusted through a trigger reconfiguration mechanism.

[0071] Example 5: This example uses 4 processing units to form a collaborative computing platform. Each processing unit interacts through a real-time data link and performs collaborative control calculations for the corresponding intelligent agent. Before the task starts, the system performs static parameter benchmark calibration, records the initial position coordinates of each intelligent agent, quantifies the sensor measurement error, and obtains the delay benchmark of the communication link.

[0072] When calibrating the communication delay, each processing unit continuously sends 100 test data packets, records the round-trip time of each data packet and calculates the average. When the average round-trip time is greater than 50ms, the system performs timing synchronization compensation to correct the clock reference offset of each agent to the synchronization range.

[0073] During task execution, the system continuously monitors the path planning vectors of each agent and judges the collision risk based on the spatial overlap between the planned paths. Path planning conflicts between agents are recorded as discrete pulse signals. The system counts the frequency of pulse signals within a 1.0s time window to obtain the objection strength, which is calculated using the following formula: ,in, For the strength of the objection, For the first The decision conflict event corresponding to each detection is set to 1 when a conflict is detected and 0 when no conflict is detected. The total number of conflict events within the time window; The statistical time window is fixed at 1.0s.

[0074] When agents A and B are performing path planning, if the controller detects that their planned paths overlap spatially, it generates a conflict event and updates the system. When the intensity of objection When the frequency exceeds 0.3Hz, the system adjusts the control weights corresponding to the conflicting agents according to the preset suppression gain coefficient. The update formula for the control weights is as follows: ,in, The updated control weight coefficients, The control weight coefficients before the update. To suppress the gain coefficient, a fixed value of 0.5 is set.

[0075] After the control weights are reduced, the response priority of the agent in the conflict area decreases accordingly. Taking agent A as an example, it adjusts its velocity vector according to the updated control weights and gradually moves out of the path overlap area with agent B. When the objection strength... When the frequency drops below 0.1 Hz, the system restores the original control weight coefficients of the intelligent agent.

[0076] The system evaluates its operational status based on the success rate of collaborative tasks. During continuous operation, when the success rate of collaborative tasks remains above 95%, each agent enters a steady-state collaborative control interval. Through the combination of a 1.0s statistical time window, a 0.5 suppression gain coefficient, and two objection intensity thresholds of 0.3Hz and 0.1Hz, the system adjusts the control weights of the agents according to the frequency of path conflicts and restores the original control weights after the conflicts decrease.

[0077] Example 6: In this example, offline parameter pre-calibration is performed to optimize the cooperative control strategy of the multi-agent system before system deployment. The offline cooperative test platform includes three homogeneous agents. Each agent is equipped with a processor capable of floating-point operations and a memory for recording cooperative behavior. The memory records the agent's output state data at a sampling frequency of 100Hz. The platform determines the initial value of the cooperative interaction weight of each agent by offline simulation of multiple typical task scenarios, and uses it as the calibration value of the decision entropy.

[0078] During offline testing, the processor continuously monitors the interaction data between the agents. When action conflicts or resource preemption are detected, these conflicts are defined by specific engineering physical and electrical criteria: in the three-dimensional physical motion space, the Euclidean distance between the instantaneous centroid coordinates of two controlled agents is less than the interference safety radius of 0.50m; or during bus data interaction, two agents simultaneously write data to the same physical address of the on-chip memory due to arbitration conflicts, and the number of consecutive bus conflicts exceeds 32 clock cycles. A mark is added to the corresponding interaction record, and it is recorded as a pulse signal. This establishes a cooperative behavior pulse signal database. The dispute intensity is calculated based on the frequency of pulse signal occurrence within a preset time window, using the following formula: ,in, For the strength of the objection, This represents the total number of pulse signals detected within a preset time window. The preset time window length.

[0079] In the communication bandwidth gradient experiment, multiple collaborative task scenarios were set up. The communication bandwidth limit for each agent to transmit action indicator feature streams was gradually adjusted from 100kbps to 1000kbps. The interval between two adjacent bandwidth levels was 100kbps. Each bandwidth level used a preset time window of 10s, and the corresponding objection intensity and collaborative control strategy execution deviation were recorded.

[0080] When the communication bandwidth is less than 300kbps, the number of action conflicts and resource preemption recorded within the preset time window increases, the objection intensity increases accordingly, and the deviation of the collaborative control strategy execution exceeds the collaborative control stability deviation threshold of 0.05. When the communication bandwidth is greater than 800kbps, the bus load of the data link increases, the average interaction delay rises, and the system collaborative response time is correspondingly extended. Within the communication bandwidth range of 400kbps to 700kbps, the objection intensity remains between 0.01Hz and 0.03Hz.

[0081] Based on the offline test results, the preferred working range of the communication bandwidth is determined to be 400kbps to 700kbps, and 0.01Hz to 0.03Hz is used as the calibration range for the objection strength. Based on this, the initial values ​​of the collaborative interaction weights of each agent are determined. Through the offline parameter pre-calibration process, interaction data such as action conflicts and resource preemption are converted into quantitative inputs for reinforcement learning policy optimization.

[0082] During system operation, the controller calculates the objection strength in real time according to a preset time window of 10 seconds. When the objection strength deviates from the calibration range of 0.01Hz to 0.03Hz, the controller triggers the reconstruction mechanism, recalibrates the collaborative interaction weights of each agent based on the currently collected interaction data, and updates the internal connection weights of the corresponding local policy network.

[0083] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A strategy optimization method for cooperative control of a multi-agent system, characterized in that, Includes the following steps: Step S101: Obtain the local raw observation data vector collected by the environmental state acquisition unit, input it into the temporal feature extraction network to complete the temporal context association calculation, and output the local state representation vector; Step S102: Obtain the action indicator feature stream received by the wireless communication interface, input it into the intent coding network to complete the nonlinear feature projection, and output the potential intent feature vector. Step S103: Input the local state representation vector and the latent intention feature vector into the mutual information estimation model to complete the calculation of the lower bound of mutual information and output the local credit allocation weight. Step S104: Calculate the advantage function based on the local task reward input from the environment, multiply correct the local policy gradient output by the local policy network using local credit allocation weights to generate the local policy update gradient, update the internal connection weights of the local policy network based on the local policy update gradient, and output the multidimensional control instruction vector to the state transition unit. A reverse adjustment loop is constructed between steps S101 and S104: the computing node reads the norm of the local policy update gradient in the last 10 cycles from memory, calculates the variance of the norm and multiplies it with the Shannon information entropy of the current action probability distribution vector of the local policy network to obtain the credit distribution dispersion factor. When the credit distribution dispersion factor is greater than 0.85, the configuration register is automatically rewritten, the sliding window history step number parameter of the temporal feature extraction network is adjusted from 3 to 8, and the received quantization bit width of the action indicator feature stream is switched from 4 bits to 16 bits.

2. The strategy optimization method for cooperative control of a multi-agent system according to claim 1, characterized in that, The calculation of the credit distribution dispersion factor includes the following steps: A fixed-length circular buffer of length 10 is allocated in the memory of the computing node; the norm of the local policy update gradient within the most recent 10 periods is read and stored cyclically; the sample variance of the norm change within the circular buffer is calculated; the Shannon information entropy of the current action probability distribution vector of the local policy network output layer is calculated; the sample variance is multiplied by the Shannon information entropy using an on-chip floating-point computing unit to output the credit distribution dispersion factor.

3. A strategy optimization method for cooperative control of a multi-agent system according to claim 1, characterized in that, In step S101, the temporal feature extraction network adopts a two-layer bidirectional gated recurrent unit network structure with 64 hidden layer neurons. Each computing node reads environmental obstacle distribution data in a loop through the synchronous serial peripheral interface and the external memory interface, and combines it with the spatial motion acceleration data of the computing node collected by the inertial navigation unit. The environmental obstacle distribution data and the spatial motion acceleration data are spliced ​​together to form a local original observation data vector.

4. A strategy optimization method for cooperative control of a multi-agent system according to claim 1, characterized in that, In step S102, each computing node receives the action indicator feature stream through the data transmission chip via the universal asynchronous transceiver interface. The action indicator feature stream is a 16-bit binary data message. The action indicator feature stream is transformed and output as a 128-dimensional latent intent feature vector by the on-chip floating-point matrix multiplier in the intent coding network to complete the nonlinear feature projection mapping.

5. A strategy optimization method for cooperative control of a multi-agent system according to claim 1, characterized in that, In step S103, the mutual information estimation model constructs a lower bound loss function for mutual information based on the contrastive learning mechanism, updates the correlation mapping weight matrix through stochastic gradient descent to maximize the correlation between positive sample pairs of local state representation vectors and potential intention feature vectors, while minimizing the correlation with negative samples of non-matching intention features randomly sampled from the experience replay area, and calculates the output local credit assignment weight.

6. A strategy optimization method for cooperative control of a multi-agent system according to claim 1, characterized in that, In step S104, limiting the boundary self-constraint range of the local policy update gradient includes the following sub-steps: Step S1041, calculate the cosine similarity between the local policy update gradient and the historical average update gradient in the last 5 periods; Step S1042, when the cosine similarity is lower than the safe similarity threshold of 0.30 and the calculated local credit allocation weight exceeds the preset range, call the projection boundary constraint operator to limit the local credit allocation weight between 0.10 and 0.30, and use the Lagrange multiplier method to project the current local policy update gradient vector back into the safe manifold space of the convex hull centered on the historical average update gradient to obtain the constrained update gradient.

7. A strategy optimization method for cooperative control of a multi-agent system according to claim 1, characterized in that, In step S104, the advantage function is calculated using the generalized advantage estimation structure based on the local reward and discount factor; the multi-dimensional control command vector obtained by the computing node includes the multi-axis motion setpoints of the distributed controlled object.

8. A strategy optimization method for cooperative control of a multi-agent system according to claim 1, characterized in that, In step S104, the multidimensional control command vector is converted by the microprocessor's timer to output a pulse width modulation signal with a variable duty cycle. The pulse width modulation signal is output to the state transformation unit through the microprocessor's physical pins to adjust the instantaneous speed of the motor at a frequency of 50Hz, thereby changing the spatial displacement state of the distributed controlled object to complete the adaptive cancellation of time-varying horizontal physical disturbances.

9. A strategy optimization method for cooperative control of a multi-agent system according to claim 1, characterized in that, The reverse adjustment loop also includes the following steps: when the credit distribution dispersion factor is continuously lower than the dispersion lower limit threshold of 0.20, the computing node rewrites the configuration register, dynamically adjusts the sliding window historical step number parameter in the time series feature extraction network from 3 to 2, and compresses the received quantization bit width of the action indicator feature stream from 16 bits to 4 bits.

Images