Trajectory tracking control method fusing multi-objective optimization and physical sensing network

Abstract

The invention relates to the technical field of intelligent control and reinforcement learning technologies, in particular to a trajectory tracking control method fusing multi-objective optimization and a physical sensing network, which comprises the following steps: acquiring a real-time state vector of a to-be-controlled object, decomposing the acquired state vector into a sphere dynamic flow and a platform attitude flow, coding features of different attitude flows are extracted, a fusion feature vector is constructed, and at the same time, an attention mechanism is used to carry out feature transformation to determine control decision features; and for the determined control decision features, utilizing a multi-objective optimization function to carry out cooperative constraint on the generated actions, carrying out feature training in combination with an experience playback mechanism and a self-adaptive stable learning mechanism, and after training is completed, determining a trajectory tracking control instruction of the object to be controlled through dynamic adjustment of learning parameters. According to the invention, by establishing an adaptive stable learning mechanism, the learning rate and exploration noise are dynamically adjusted based on performance stagnation detection, and the training stability and convergence speed are improved.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent control and reinforcement learning technology, specifically to a trajectory tracking control method that integrates multi-objective optimization and physical perception networks. Background Technology

[0002] With the rapid development of intelligent manufacturing and automation technologies, the demand for trajectory tracking control of complex nonlinear systems is increasing, such as the balance control of ball-plate systems, the precision operation of robotic arms, and the path tracking of mobile robots. These controlled objects typically have characteristics such as strong nonlinearity, high-dimensional states, and complex coupling. Traditional methods such as PID control and LQR control rely on accurate dynamic models, and their performance is limited when faced with modeling errors, external disturbances, or complex trajectory tasks.

[0003] In recent years, control methods based on deep reinforcement learning, such as DDPG and TD3, have achieved certain results in continuous control tasks. However, existing deep reinforcement learning control frameworks have the following main problems: a single reward design cannot simultaneously cover multiple performance requirements such as trajectory accuracy, smoothness, and stability, resulting in unclear training objectives; the training process is prone to oscillations or stagnation, lacking a stable training mechanism; the experience playback mechanism does not consider the differences in training stages, and a large number of low-value samples interfere with training efficiency; the policy network does not fully utilize the physical structure information of the control system and cannot effectively extract key features of sphere dynamics and platform attitude, resulting in unstable action output and frequent oscillations.

[0004] Therefore, a deep reinforcement learning control framework with higher interpretability, more stable training process, and better performance is needed to achieve high-precision, low-oscillation, and robust trajectory tracking control. Summary of the Invention

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a trajectory tracking control method integrating multi-objective optimization and physical sensing networks, comprising the following steps:

[0006] The real-time state vector of the object to be controlled is obtained, and the obtained state vector is decomposed into sphere dynamics flow and platform attitude flow. The encoded features of different attitude flows are extracted and a fused feature vector is constructed. At the same time, the attention mechanism is used to perform feature transformation to determine the control decision features.

[0007] Furthermore, for the determined control decision characteristics, a multi-objective optimization function is used to coordinately constrain the generated actions, and feature training is carried out by combining an experience playback mechanism and an adaptive stable learning mechanism. After training, the trajectory tracking control command of the object to be controlled is determined by dynamically adjusting the learning parameters.

[0008] As a preferred embodiment of the trajectory tracking control method integrating multi-objective optimization and physical sensing network described in this invention, the construction of the fused feature vector is specifically as follows:

[0009] The object to be controlled is set as a ball-plate series of objects. The real-time state vector is obtained and, based on the physical coupling relationship between the object to be controlled and the controller, the 8-dimensional real-time state vector is split into two types of feature flows, including the ball dynamics flow and the platform attitude flow.

[0010] For the split feature stream, feature extraction is performed separately using an encoder, and the extracted encoded features are then fused and concatenated.

[0011] Based on the extracted encoded features, the features are concatenated according to the dimensions to form a fused feature vector.

[0012] As a preferred embodiment of the trajectory tracking control method integrating multi-objective optimization and physical perception network described in this invention, the step of using an attention mechanism to perform feature transformation to determine control decision features is as follows:

[0013] Based on the fused feature vector, the query matrix, key matrix, and value matrix of the fused feature vector are obtained through linear transformation;

[0014] Based on a defined query matrix, key matrix, and value matrix, a multi-head self-attention mechanism is used to capture feature associations in different dimensions and obtain the final fused features.

[0015] Actions are determined based on fused feature vectors, and the determined actions are mapped to 4-dimensional action vectors that include PID parameter adjustments.

[0016] As a preferred embodiment of the trajectory tracking control method integrating multi-objective optimization and physical sensing network described in this invention, the step of using a multi-objective optimization function to perform cooperative constraints on the generated actions is as follows:

[0017] A multi-objective reward function is constructed based on trajectory error, control smoothness, system stability, tracking robustness, phase error, completion rate, and oscillation indicators.

[0018] Meanwhile, the multi-objective reward function is constructed by a weighted combination of penalty and reward indicators, and the weights of each indicator in the multi-objective reward function are dynamically adjusted according to the training stage and performance history.

[0019] The effect of each control step is evaluated based on the set multi-objective reward function, and an optimized control strategy is executed based on the evaluation results.

[0020] As a preferred embodiment of the trajectory tracking control method integrating multi-objective optimization and physical sensing network described in this invention, the experience playback mechanism is specifically as follows:

[0021] A phase-aware priority experience replay mechanism is adopted to train and optimize the multi-objective optimization function, including calculating the priority of experience samples based on temporal difference error; introducing a reward threshold for sample filtering in the later stage of training; and correcting the priority sampling bias through importance sampling weight.

[0022] As a preferred embodiment of the trajectory tracking control method integrating multi-objective optimization and physical sensing network described in this invention, the adaptive stabilization learning mechanism is specifically as follows:

[0023] It is achieved through a three-step closed loop: performance monitoring, state determination, and parameter adjustment; including storing the average reward of recent training rounds using a fixed-capacity sliding window to quantitatively evaluate training performance.

[0024] Determine if performance has stagnated by the difference in the average reward between the previous and subsequent stages;

[0025] Furthermore, the learning rate and exploration noise are adjusted for three states: stagnation, continuous stagnation, and normal training.

[0026] As a preferred embodiment of the trajectory tracking control method integrating multi-objective optimization and physical sensing network described in this invention, the determination of the trajectory tracking control command of the object to be controlled through dynamic adjustment of learning parameters is as follows:

[0027] Stable training is achieved by maintaining multiple networks. The process is executed in four steps: target action generation, target Q-value calculation, network loss solution, and parameter update. This determines the trajectory tracking control command for the object to be controlled. The multi-target network includes a policy network and a value network.

[0028] As a preferred embodiment of the trajectory tracking control method integrating multi-objective optimization and physical sensing network described in this invention, the network loss solution is specifically as follows:

[0029] Loss calculation and parameter updates are performed on the target policy network and the target value network respectively, including:

[0030] The loss calculation and parameter update of the target value network are to minimize the mean square error between the current Q-value and the target Q-value, for a batch size of... Using the samples, construct the loss function;

[0031] The loss calculation and parameter update of the target policy network adopt an asynchronous update strategy, that is, every 2 value network updates correspond to 1 policy network update;

[0032] The target network is updated by slowly synchronizing the parameters of the current network using an update strategy.

[0033] As a preferred embodiment of the trajectory tracking control method integrating multi-objective optimization and physical perception network described in this invention, the quantitative evaluation of training performance is specifically as follows:

[0034] Set capacity A sliding window stores the average reward of the last 50 training rounds in real time, and the window is updated using the first-in-first-out principle.

[0035] Divide the 50 data points within the sliding window into two segments, and calculate the difference in average reward between the two segments as the performance improvement. The formula is:

[0036]

[0037] in, This indicates a performance improvement, and vice versa. This indicates the average reward.

[0038] The beneficial effects of this invention are:

[0039] By constructing a multi-objective performance-driven reward mechanism, multiple performance indicators such as trajectory error, control smoothness, system stability, and oscillation index are weighted and combined, and the weights are dynamically adjusted according to the training phase. This effectively solves the problem that a single reward design cannot cover multiple performance requirements at the same time, and achieves the coordinated optimization of multiple objective performances such as accuracy, stability, and smoothness.

[0040] By establishing an adaptive stable learning mechanism, the learning rate and exploration noise are dynamically adjusted based on performance stagnation detection, which effectively solves the problem of oscillation or stagnation that easily occurs during the training process and significantly improves training stability and convergence speed.

[0041] By designing a priority-aware experience replay mechanism, prioritization is calculated based on temporal differential error, and filtering is performed according to the reward threshold in the later stages of training. This effectively improves sample utilization by 30-60% and avoids interference from a large number of low-value samples in training.

[0042] By employing a physical perception Transformer policy network, the state vector is decomposed into sphere dynamics flow and platform attitude flow. Feature fusion is performed through dual encoders and Transformer attention mechanism, which makes full use of the physical structure information of the control system, significantly enhances the perception capability and interpretability of the policy network, makes the action output more stable, and reduces the oscillation frequency by more than 40%. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0044] Figure 1 This is a schematic diagram of the overall method steps of the trajectory tracking control method that integrates multi-objective optimization and physical sensing network according to the present invention. Detailed Implementation

[0045] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0046] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0047] Example 1

[0048] Reference Figure 1 This is the first embodiment of the present invention, which provides a trajectory tracking control method that integrates multi-objective optimization and physical sensing networks, including the following steps:

[0049] S1: Obtain the real-time state vector and determine the control decision characteristics.

[0050] Specifically, the real-time state vector of the object to be controlled is acquired and decomposed into sphere dynamics flow and platform attitude flow. Features are extracted from the sphere dynamics flow using a first encoder and from the platform attitude flow using a second encoder. The two encoded features are concatenated and input into the Transformer encoder. The learned attention weights adaptively capture the correlation information between different physical flows to generate fused features. The fused features are then mapped into a 4D action vector by the action generation unit, and the PID parameter adjustment is output. The specific implementation is as follows:

[0051] The object to be controlled is set to a ball-and-plate series of objects, and the real-time state vector obtained is... Then there is,

[0052]

[0053] in, , They represent the sphere in , Positional deviation of direction , These represent the sphere's velocity, , These represent the platform's roll angle and pitch angle, respectively. , These represent the platform's angular velocity.

[0054] Based on the physical coupling relationship between the controlled object and the controller, the 8-dimensional real-time state vector is decomposed into two types of feature streams, specifically:

[0055] The dynamic flow of a sphere, which includes information on the sphere's positional deviation and velocity, then we have:

[0056]

[0057] The platform attitude flow, which contains the platform's attitude angles and angular velocity information, then we have:

[0058]

[0059] in, , They represent the sphere in , Positional deviation of direction , These represent the sphere's velocity, , These represent the platform's roll angle and pitch angle, respectively. , These represent the platform angular velocities, Represents the dynamic flow of a sphere. This represents the platform's attitude flow.

[0060] For the split feature streams, features are extracted separately using encoders, and the extracted encoded features are then fused and concatenated. The specific implementation is as follows:

[0061] If features are extracted from the spherical dynamics using the first encoder, then we have:

[0062]

[0063] in, Represents the dynamic flow of a sphere. This represents the activation function. This represents normalization, used to suppress data bias during feature extraction. (Input dimension → Output dimension) represents a fully connected layer. To prevent overfitting and random deactivation, the specific operation should be set by the implementer based on the actual application scenario. This represents the final output sphere dynamics encoding features;

[0064] If features are extracted from the platform pose flow using a second encoder, then we have:

[0065]

[0066] in, Indicates platform posture flow, This represents the activation function. This represents normalization, used to suppress data bias during feature extraction. (Input dimension → Output dimension) represents a fully connected layer. To prevent overfitting and random deactivation, the specific operation should be set by the implementer based on the actual application scenario. This represents the final output platform attitude coding features.

[0067] Based on the extracted encoded features, the features are concatenated according to their dimensions to form a fused feature vector, then we have:

[0068]

[0069] in, This indicates the final output sphere dynamics encoding features. This represents the final output platform pose encoding features. This represents the fused feature vector, which is a 256-dimensional fused feature.

[0070] Based on the fused feature vectors, the Transformer attention mechanism is used to transform the features, thereby determining the control decision features, as follows:

[0071] Based on the fused feature vectors, the query matrix, key matrix, and value matrix of the fused feature vectors are obtained through linear transformation. Then, we have...

[0072]

[0073]

[0074]

[0075] in, , , These represent the weight matrix, which is set by the implementer according to the actual application scenario and is used to capture the correlation information between features. Represents the query matrix. Represents the key matrix. Represents a value matrix, This represents the fused feature vector;

[0076] Based on a defined query matrix, key matrix, and value matrix, a multi-head self-attention mechanism is used to capture feature associations across different dimensions. Therefore,

[0077]

[0078] in, This represents the key dimension vector, used to scale the attention weights. This represents the transpose of the key matrix. This represents the normalization function, ensuring that the sum of the attention weights is 1. This indicates the number of multi-head attention sessions, each capturing feature associations across different dimensions. Represents the query matrix. Represents the key matrix. Represents a value matrix;

[0079] The multi-head attention output, after passing through residual connections (preserving original feature information) and layer normalization (for stable training), undergoes further feature transformation via a feedforward network. After passing through several Transformer encoding layers, the final fused features are obtained. .

[0080] The fusion features output by the Transformer attention mechanism The input is fed into the action generation unit and mapped to the adjustment amount of the PID parameters for actual control. The specific implementation is as follows:

[0081] Based on the fused feature vectors, the action is determined, and the determined action is mapped to the PID parameter adjustment amount. Then, we have...

[0082] Based on the network structure to determine the action, we have:

[0083]

[0084] in, Indicates fusion features, , This represents the activation function. This represents normalization, used to suppress data bias during feature extraction. (Input dimension → Output dimension) represents a fully connected layer. To prevent overfitting and random deactivation, the specific operation should be set by the implementer based on the actual application scenario. Indicates a definite action;

[0085] Mapping a defined action to a 4-dimensional action vector that includes PID parameter adjustments, we have:

[0086]

[0087] in, , , and represent the normalized adjustment values ​​of the proportional, integral, and derivative parameters of the PID controller, respectively. This represents the normalized adjustment amount of the feedforward gain. This represents the output range of the Tanh activation function. The specific value is set by the implementer according to the actual application scenario. This embodiment is for illustrative purposes only and does not impose any limitations. Indicates a definite action.

[0088] S2: Based on the generated motion, determine the trajectory tracking control command of the object to be controlled through the drive control strategy.

[0089] Specifically, determining the trajectory tracking control command of the object to be controlled through a drive control strategy involves using a multi-objective optimization function to collaboratively constrain the generated actions, and combining experience playback and adaptive stable learning mechanisms to dynamically adjust the learning parameters, thereby determining the trajectory tracking control command of the object to be controlled. The specific implementation is as follows:

[0090] A multi-objective reward function is constructed based on trajectory error, control smoothness, system stability, tracking robustness, phase error, completion rate, and oscillation indicators. This function is formed by a weighted combination of penalty and reward indicators, and the weights of each indicator are dynamically adjusted according to the training stage and performance history. Specifically:

[0091]

[0092] in, , , , , , , These represent the weighting coefficients for trajectory error, control smoothness, system stability, tracking robustness, phase error, completion rate, and oscillation index, respectively. , , , , These represent the degradation indicators that need to be penalized (the higher the value, the lower the reward). , These represent the excellent indicators that need to be rewarded (the higher the value, the higher the reward).

[0093] Trajectory Error Index It uses a fixed window size. Calculate and collect the first Given the actual trajectory and the target trajectory at each sampling time, we have:

[0094] Set within the window The actual trajectory at each sampling time is The target trajectory is Then there is,

[0095]

[0096] in, Indicates the first in the window The actual trajectory at each sampling time Indicates the first in the window The target trajectory at each sampling time Indicator representing the calculated trajectory error;

[0097] Control smoothness index It is based on time intervals Calculate the rate of change of control action and set For the first The action vector at each sampling time point then has,

[0098]

[0099] in, This indicates the set time interval. Indicates the first Action vector at each sampling time, This indicates the control smoothness index of the calculation;

[0100] Saturation index It is used to measure the proportion of the control quantity entering the saturation range, and the control quantity vector is set as... , its first The upper limit of dimensional physics is Define a single-step saturation indicator function:

[0101]

[0102] Then in length of The saturation index within the window is:

[0103]

[0104] The smaller the indicator value, the closer the control quantity is to the normal working range;

[0105] Phase error index This is used to measure the phase lag between the target trajectory and the actual response. Let a certain dimension of error be... The time delay that maximizes the cross-correlation function is obtained through cross-correlation analysis. :

[0106]

[0107] The phase error index is defined as the absolute value of the normalized time delay:

[0108]

[0109] in, The length of the window;

[0110] For a two-dimensional trajectory, the x and y dimensions can be averaged or summed. This embodiment uses the two-dimensional average for illustration, then we have:

[0111]

[0112] Oscillators Taking into account the oscillation frequency, oscillation amplitude, and error fluctuation, we have:

[0113]

[0114] The oscillation frequency is the normalized result of the number of error sign changes (zero crossovers) per unit time. This represents the average or maximum value of the peak-to-trough difference in error within the window, after normalization. This represents the standard deviation of the error sequence within the window, after normalization.

[0115] In terms of belt rate It reflects whether the trajectory error remains stably within the allowable error band. Let the error magnitude be:

[0116]

[0117] Let the allowable error band radius be Define the indicator function:

[0118]

[0119] The carrier belt ratio index is:

[0120]

[0121] Completion Index If it is used to measure the percentage of the target trajectory completed, then we have:

[0122] Set the total trajectory length of the target as The effective length of the currently tracked trajectory is (For example, the cumulative arc length of the actual trajectory within the allowable error band), then:

[0123]

[0124] For local rounds, an approximation can be made based on the relative completion rate within the executed time.

[0125] The weights of each metric are dynamically adjusted based on the training phase and performance history, specifically as follows:

[0126] set up And satisfy the formula Dynamic adjustment, then,

[0127] Early training phase ( ):

[0128]

[0129] Mid-training ( ):

[0130]

[0131] Later stages of training ( ):

[0132]

[0133] Other weights In implementation, the settings are determined by the implementers based on the actual application scenario.

[0134] It should be noted that the control environment evaluates the control effect at each step based on the set multi-objective reward function: if the system trajectory error decreases, the control is smooth, and oscillations are suppressed, then... The value is relatively large; if the system exhibits divergence, increased oscillation, or control saturation, then... If the value is negative, an optimization control strategy will be executed.

[0135] Furthermore, the experience replay mechanism employs a stage-aware priority experience replay mechanism to train and optimize the multi-objective optimization function. This includes calculating the priority of experience samples based on temporal difference errors to ensure that samples are sampled; introducing a reward threshold for sample filtering in the later stages of training; and correcting priority sampling bias through importance sampling weights to avoid training deviation. The specific implementation is as follows:

[0136] Priority calculation and sampling are implemented as follows:

[0137] Define the experience sample as ,in For the current state, To perform the action, For reward feedback, For the next state,

[0138] Calculate sample priority And related parameters, then,

[0139] Priority is determined by temporal difference error (TD error). The larger the TD error, the more new information the sample contains, and the higher the priority. The formula is:

[0140]

[0141] in, This represents a very small constant, used to prevent a sample's TD error from being zero, which would result in a priority of 0, ensuring that all samples have the potential to be sampled. This represents the priority weighting coefficient, used to control the bias in priority sampling. Degenerate into uniform sampling, Pure priority sampling; This represents the time-series difference error, used in the computation of dual-Critic networks employing the TD3 algorithm to improve stability and avoid overestimation;

[0142] Calculating the timing difference error, we have:

[0143] The TD error is calculated collaboratively using a dual-objective Critic network and an objective Actor network, as shown in the formula:

[0144]

[0145] in, This represents a discount factor used to weigh the importance of current rewards against future long-term rewards. Indicates the first A target Critic network (with fixed parameters to avoid training oscillations) is used to output the action value of a state. This represents the optimal action for the next time step, as output by the target Actor network. This represents the action value of the current state output by the Critic network.

[0146] Introducing a reward threshold for sample filtering in the later stages of training involves filtering samples by introducing a reward threshold during the mid-to-late training phase. Specifically:

[0147] Set the 60th percentile of the reward for all samples in the current experience pool as the reward threshold, and filter out sample data that exceeds the reward threshold based on the set reward threshold.

[0148] It should be noted that priority sampling can lead to sample distribution skew (high-priority samples are oversampled), so importance sampling weights need to be introduced. If we adjust the loss function to ensure unbiased training, then we have:

[0149]

[0150] in, This represents the total capacity of the experience replay pool. Indicates the first The normalized sampling probability of each sample. This represents the bias compensation coefficient, which increases linearly from 0.4 at the beginning of training to 1.0 at the end of training—initial weakening correction (allowing a certain bias), and later full compensation (ensuring unbiased training).

[0151] Furthermore, the adaptive stable learning mechanism monitors training performance in real time through a sliding window, dynamically adjusting the learning rate of the policy network and the standard deviation of the exploration noise. This achieves an adaptive training effect that ensures stable convergence when performance improves and promotes breakthroughs when performance stagnates. The specific implementation is as follows:

[0152] The adaptive stable learning mechanism is achieved through a three-step closed loop: performance monitoring, state determination, and parameter adjustment. This includes: storing the average reward of recent training rounds using a fixed-capacity sliding window to quantitatively evaluate training performance; determining whether performance has stagnated by the difference in the average reward between preceding and following stages; and adjusting the learning rate and exploration noise for three states: stagnation, continuous stagnation, and normal training, respectively, to balance convergence stability and exploration diversity. Specifically:

[0153] Quantitative evaluation of training performance uses the average reward per round as the core metric, and achieves dynamic monitoring through a sliding window and difference calculation, specifically as follows:

[0154] Maintenance capacity A sliding window that stores the average reward of the most recent 50 training rounds in real time. ( , (For the latest round), the window updates using a first-in, first-out (FIFO) principle to ensure data timeliness;

[0155] Divide the 50 data points within the sliding window into two segments (25 rounds each), and calculate the difference in average reward between the two segments as the performance improvement. The formula is:

[0156]

[0157] in, This indicates a performance improvement, and vice versa. This indicates the average reward.

[0158] Determining whether performance has stagnated by comparing the average reward before and after a stage is done by setting a performance stagnation threshold, specifically:

[0159] Set performance stagnation threshold When the formula is satisfied If the training performance is stagnant, a parameter adjustment strategy is triggered; otherwise, it is a normal training state.

[0160] The learning rate and exploration noise are adjusted adaptively based on the network's performance status (normal, stagnant, or continuously stagnant). The specific adjustments are as follows:

[0161] When performance stagnates, parameter adjustments are made by "increasing the learning rate and increasing exploration noise" to stimulate policy diversity and help escape local optima. Specifically:

[0162] If the learning rate of the policy network increases, then...

[0163]

[0164] Learning rate Increased to 1.3 times, while the upper limit of the constraint is... To avoid excessively high learning rates that could cause training oscillations;

[0165] If the standard deviation of the noise increases, then...

[0166]

[0167] Increase the noise level by 0.05 based on the current noise level, with a constraint upper limit of 0.4, to ensure that the exploration range is controllable.

[0168] The parameter adjustments during periods of continuous performance stagnation are adjustments made for periods of stable convergence, specifically as follows:

[0169] If the performance stagnation continues If there is no performance improvement within 20 rounds (i.e., 20 rounds), it is determined that the policy update direction is biased. To stabilize training by reducing the learning rate, we have:

[0170]

[0171] Reduce the learning rate to 0.7x, and the lower bound of the constraint is... This is to prevent training from stagnating due to an excessively low learning rate.

[0172] The parameter adjustments during normal training are for steady-state convergence, specifically:

[0173] If no performance stagnation is detected, it indicates that the current strategy update direction is effective. By "gradually reducing exploration noise" to decrease invalid explorations and accelerate convergence, we have:

[0174]

[0175] Each round, the noise standard deviation is reduced by 0.001, with a lower limit of 0.05, preserving basic exploration capabilities to cope with minor environmental changes.

[0176] Furthermore, determine the trajectory tracking control command for the object to be controlled.

[0177] Stable training is achieved by maintaining multiple networks, following a four-step process: target action generation, target Q-value calculation, network loss solution, and parameter update. This process determines the trajectory tracking control commands for the object to be controlled. The multiple networks include a policy network and a value network. Each network group contains the current network and the target network. The specific implementation is as follows:

[0178] Smooth noise is added to the actions output by the target policy network, and action clipping is performed, as follows:

[0179] Adding smoothed noise, we have:

[0180]

[0181] in, The standard deviation of noise. For the target policy network, the next state The original action output;

[0182] Motion clipping, then,

[0183]

[0184] Among them, the range of motion is constrained as follows: This ensures that the action is within the effective range of the PID parameters of the ball plate system, avoiding extreme values ​​that could cause system instability.

[0185] The objective Q-value calculation involves finding the minimum output of the bi-objective value network, specifically:

[0186]

[0187] in, This represents the immediate reward in the empirical sample (from a multi-objective performance-driven reward mechanism). Indicates the reward discount factor; This indicates taking the minimum value output of the two target value networks.

[0188] The network loss calculation involves calculating the loss and updating the parameters for both the target policy network and the target value network. Specifically:

[0189] The loss calculation and parameter update of the target value network are to minimize the mean square error between the current Q-value and the target Q-value, for a batch size of... For the samples, construct the loss function, then we have,

[0190]

[0191] And by updating the constructed loss function, we have:

[0192] For the current Critic network , Calculate the losses mentioned above separately, and update the parameters of the two networks using gradient descent. , Each parameter update is recorded as one training step of the target value network.

[0193] The loss calculation and parameter update of the target policy network adopt an asynchronous update strategy, that is, every two value network updates correspond to one policy network update, specifically:

[0194]

[0195] And by updating the constructed loss function, we have:

[0196] After negating the loss function, minimize it using gradient descent. This allows for maximizing the value of the target policy network's output actions and using the target policy network's output to calculate the loss, thus avoiding update oscillations caused by conflicts between the two target policy networks.

[0197] It should be noted that, to ensure training stability, the target network is not updated directly; instead, the parameters of the current network are slowly synchronized using a soft update strategy. Specifically:

[0198] If the target policy network is updated, then...

[0199]

[0200] If the target value network is updated, then...

[0201]

[0202] in, This is the soft update coefficient. A value close to 0 ensures that the target network parameters change smoothly, providing a stable target benchmark for training.

[0203] It should be noted that the dual-target network plus target action noise-adding mechanism suppresses Q-value overestimation from both value assessment and action generation dimensions; the asynchronous update of the value network and the policy network avoids training oscillations caused by mutual interference between the two, and the soft update of the value network provides a stable benchmark for parameter optimization; together with "priority experience playback" and "adaptive stable learning", a complete training closed loop of "utilization → dynamic parameter adjustment → stable network optimization" is formed, which ultimately outputs a high-precision and high-stability control strategy adapted to the ball board system.

[0204] Furthermore, if the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0205] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0206] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0207] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A trajectory tracking control method integrating multi-objective optimization and physical sensing networks, characterized in that: Includes the following steps, The real-time state vector of the object to be controlled is obtained, and the obtained state vector is decomposed into sphere dynamics flow and platform attitude flow. The encoded features of different attitude flows are extracted and a fused feature vector is constructed. At the same time, the attention mechanism is used to perform feature transformation to determine the control decision features. Furthermore, for the determined control decision characteristics, a multi-objective optimization function is used to coordinately constrain the generated actions, and feature training is carried out by combining an experience playback mechanism and an adaptive stable learning mechanism. After training, the trajectory tracking control command of the object to be controlled is determined by dynamically adjusting the learning parameters.

2. The trajectory tracking control method integrating multi-objective optimization and physical sensing network as described in claim 1, characterized in that: The construction of the fused feature vector is as follows: The object to be controlled is defined as a ball-and-plate series of objects. The acquired real-time state vector is then decomposed into two types of feature streams based on the physical coupling relationship between the object and the controller. This includes spherical dynamics flow and platform attitude flow; For the split feature stream, feature extraction is performed separately using an encoder, and the extracted encoded features are then fused and concatenated. Based on the extracted encoded features, the features are concatenated according to the dimensions to form a fused feature vector.

3. The trajectory tracking control method integrating multi-objective optimization and physical sensing network as described in claim 2, characterized in that: The process of using an attention mechanism to transform features and determine control decision features is as follows: Based on the fused feature vector, the query matrix, key matrix, and value matrix of the fused feature vector are obtained through linear transformation; Based on a defined query matrix, key matrix, and value matrix, a multi-head self-attention mechanism is used to capture feature associations in different dimensions and obtain the final fused features. Actions are determined based on fused feature vectors, and the determined actions are mapped to 4-dimensional action vectors that include PID parameter adjustments.

4. The trajectory tracking control method integrating multi-objective optimization and physical sensing network as described in claim 3, characterized in that: The process of applying collaborative constraints to the generated actions using a multi-objective optimization function is as follows: A multi-objective reward function is constructed based on trajectory error, control smoothness, system stability, tracking robustness, phase error, completion rate, and oscillation indicators. Meanwhile, the multi-objective reward function is constructed by a weighted combination of penalty and reward indicators, and the weights of each indicator in the multi-objective reward function are dynamically adjusted according to the training stage and performance history. The effect of each control step is evaluated based on the set multi-objective reward function, and an optimized control strategy is executed based on the evaluation results.

5. The trajectory tracking control method integrating multi-objective optimization and physical sensing network as described in claim 4, characterized in that: The experience replay mechanism is as follows: A stage-aware priority experience replay mechanism is used to train and optimize the multi-objective optimization function, including calculating the priority of empirical samples based on temporal difference error; In the later stages of training, a reward threshold is introduced for sample filtering; and priority sampling bias is corrected by using importance sampling weights.

6. The trajectory tracking control method integrating multi-objective optimization and physical sensing network as described in claim 5, characterized in that: The adaptive stable learning mechanism is as follows: It is achieved through a three-step closed loop: performance monitoring, status determination, and parameter adjustment. This includes using a fixed-capacity sliding window to store the average reward from recent training rounds, thus quantitatively evaluating training performance; Determine if performance has stagnated by the difference in the average reward between the previous and subsequent stages; Furthermore, the learning rate and exploration noise are adjusted for three states: stagnation, continuous stagnation, and normal training.

7. The trajectory tracking control method integrating multi-objective optimization and physical sensing network as described in claim 6, characterized in that: The process of determining the trajectory tracking control command for the object to be controlled by dynamically adjusting the learned parameters is as follows: Stable training is achieved by maintaining multiple networks. The process is executed in four steps: target action generation, target Q-value calculation, network loss solution, and parameter update. This determines the trajectory tracking control command for the object to be controlled. The multi-target network includes a policy network and a value network.

8. The trajectory tracking control method integrating multi-objective optimization and physical sensing network as described in claim 7, characterized in that: The network loss is calculated as follows: Loss calculation and parameter updates are performed on the target policy network and the target value network respectively, including: The loss calculation and parameter update of the target value network are to minimize the mean square error between the current Q-value and the target Q-value, for a batch size of... Using the samples, construct the loss function; The loss calculation and parameter update of the target policy network adopt an asynchronous update strategy, that is, every 2 value network updates correspond to 1 policy network update; The target network is updated by slowly synchronizing the parameters of the current network using an update strategy.

9. The trajectory tracking control method integrating multi-objective optimization and physical sensing network as described in claim 8, characterized in that: The quantitative evaluation of training performance is as follows: Set capacity A sliding window stores the average reward of the last 50 training rounds in real time, and the window is updated using the first-in-first-out principle. Divide the 50 data points within the sliding window into two segments, and calculate the difference in average reward between the two segments as the performance improvement. The formula is: in, This indicates a performance improvement, and vice versa. This indicates the average reward.

Images