Design method of lightweight distributed reinforcement learning training platform based on Ray

Abstract

The present invention discloses a design method of a lightweight distributed reinforcement learning training platform based on Ray, including obtaining a set of original data streams, constructing a topological feature representation, performing dynamic smoothing, and generating a stabilized feature representation; calculating information retention and topological retention, constructing a state manifold, a tangent space, and a priority queue, and updating an experience pool; collecting strategy parameters and gradients, calculating the Karcher mean and performing momentum correction, constructing a parameter manifold hierarchical structure and inter-layer mapping, and optimizing the parameter structure; constructing a knowledge graph, performing structured pruning and quantitative optimization, and obtaining the final compression model parameters; constructing a load distribution matrix, generating an initial parameter distribution scheme and performing iterative optimization, constructing a load balancing objective function, generating a task scheduling strategy, and finally deploying the configuration. The present invention effectively reduces the complexity of data processing, improves computing efficiency, and reduces the need for manual adjustment by developers through a variety of automated optimizations.

Description

Technical Field

[0001] The present application relates to the field of reinforcement learning, and in particular to a design method for a lightweight distributed reinforcement learning training platform based on Ray. Background Art

[0002] With the rapid development of science and technology and the gradual popularization of big data technology, machine learning, especially reinforcement learning, has become a key technology for solving core problems in many fields such as industrial automation, financial engineering, and self-driving cars. These applications often involve complex decision-making processes and dynamic environmental adaptation, requiring the system to not only have efficient learning capabilities, but also to respond in real time and process large-scale data.

[0003] Traditional stand-alone learning frameworks, due to their limited architecture and computing power, are unable to meet the high requirements of modern reinforcement learning tasks for data processing capabilities and computing speed. In this regard, although modern distributed reinforcement learning frameworks have initially solved the problem, they often face a series of new challenges such as low execution efficiency, high resource usage, difficulty in system expansion, and high requirements for developer skills. These challenges limit the wider application and popularization of reinforcement learning technology.

[0004] Therefore, further research and innovation are needed to solve the above problems existing in the prior art. Summary of the invention

[0005] The purpose of the invention is to propose a design method for a lightweight distributed reinforcement learning training platform based on Ray to solve the above-mentioned problems existing in the prior art.

[0006] The technical solution is a design method for a lightweight distributed reinforcement learning training platform based on Ray, which includes the following steps:

[0007] S1. Obtain the original data stream set, construct the time-series adjacency matrix, generate the manifold embedding mapping, and obtain a unified topological feature representation; represent the topological features, calculate the feature stability matrix and stability weight, and perform dynamic smoothing to obtain a stabilized feature representation; based on the stabilized feature representation, calculate the local probability distribution and the distance-preserving metric to generate a compressed feature representation; based on the compressed feature representation and the topological feature representation, calculate the information retention rate and the topological retention degree, and generate a quality score and optimization suggestions; wherein the original data stream set includes timestamps and feature vectors;

[0008] S2. Based on compressed feature representation and quality score, construct state manifold and tangent space, calculate value function gradient and geodesic, and generate structured reward function; based on pre-stored node strategy set and structured reward function, calculate Wasserstein metric and strategy gradient and perform collaborative update to obtain optimized strategy set; based on optimized strategy set and pre-stored historical experience pool, calculate trajectory entropy and importance weight, construct priority queue, and obtain updated experience pool; based on updated experience pool, collect strategy parameters and gradient of each node, calculate Karcher mean and perform momentum correction to obtain synchronized global parameters;

[0009] S3. Based on the synchronized global parameters, construct the parameter manifold hierarchical structure and inter-layer mapping, and optimize the inter-layer connection to obtain the optimized parameter structure; based on the optimized parameter structure and the pre-stored historical model set, construct the knowledge graph, calculate the importance of knowledge units and perform selective knowledge update to obtain the updated knowledge base; obtain the current environmental features, and based on the current environmental features and the updated knowledge base, construct the environmental feature manifold and invariance measurement to generate a generalization strategy; Based on the generalization strategy, calculate the parameter sensitivity and importance and perform structured pruning and quantization optimization to obtain the final compression model parameters;

[0010] S4. Based on the final compression model parameters and pre-stored performance indicators, a load distribution matrix is ​​constructed to generate an initial parameter distribution scheme; the initial parameter distribution scheme is iteratively optimized to obtain an optimized distribution scheme; based on the optimized distribution scheme, a load balancing objective function is constructed to generate a task scheduling strategy; based on the task scheduling strategy, a final deployment configuration is generated.

[0011] Beneficial effects: The present invention can effectively reduce the complexity of data processing and improve computing efficiency by constructing a temporal adjacency matrix, manifold embedding mapping and compressed feature representation, while also reducing the resource usage of data storage and computing; through the calculation of distributed nodes and synchronized global parameters, it helps to achieve horizontal expansion of the system; and through a variety of automated optimizations, it reduces the need for manual adjustments by developers. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flow chart of the present invention.

[0013] Figure 2 This is a flow chart of step S1 of the present invention.

[0014] Figure 3 This is a flow chart of step S2 of the present invention.

[0015] Figure 4 This is a flow chart of step S3 of the present invention.

[0016] Figure 5This is a flow chart of step S4 of the present invention. DETAILED DESCRIPTION

[0017] like Figure 1 As shown, the present invention proposes a design method for a lightweight distributed reinforcement learning training platform based on Ray, comprising the following steps:

[0018] S1. Obtain the original data stream set, construct the time-series adjacency matrix, generate the manifold embedding mapping, and obtain a unified topological feature representation; represent the topological features, calculate the feature stability matrix and stability weight, and perform dynamic smoothing to obtain a stabilized feature representation; based on the stabilized feature representation, calculate the local probability distribution and the distance-preserving metric to generate a compressed feature representation; based on the compressed feature representation and the topological feature representation, calculate the information retention rate and the topological retention degree, and generate a quality score and optimization suggestions; wherein the original data stream set includes timestamps and feature vectors;

[0019] S2. Based on compressed feature representation and quality score, construct state manifold and tangent space, calculate value function gradient and geodesic, and generate structured reward function; based on pre-stored node strategy set and structured reward function, calculate Wasserstein metric and strategy gradient and perform collaborative update to obtain optimized strategy set; based on optimized strategy set and pre-stored historical experience pool, calculate trajectory entropy and importance weight, construct priority queue, and obtain updated experience pool; based on updated experience pool, collect strategy parameters and gradient of each node, calculate Karcher mean and perform momentum correction to obtain synchronized global parameters;

[0020] S3. Based on the synchronized global parameters, construct the parameter manifold hierarchical structure and inter-layer mapping, and optimize the inter-layer connection to obtain the optimized parameter structure; based on the optimized parameter structure and the pre-stored historical model set, construct the knowledge graph, calculate the importance of knowledge units and perform selective knowledge update to obtain the updated knowledge base; obtain the current environmental features, and based on the current environmental features and the updated knowledge base, construct the environmental feature manifold and invariance measurement to generate a generalization strategy; Based on the generalization strategy, calculate the parameter sensitivity and importance and perform structured pruning and quantization optimization to obtain the final compression model parameters;

[0021] S4. Based on the final compression model parameters and pre-stored performance indicators, a load distribution matrix is ​​constructed to generate an initial parameter distribution scheme; the initial parameter distribution scheme is iteratively optimized to obtain an optimized distribution scheme; based on the optimized distribution scheme, a load balancing objective function is constructed to generate a task scheduling strategy; based on the task scheduling strategy, a final deployment configuration is generated.

[0022] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0023] S11. Obtain a set of original data streams D = {d1, d2, ..., dn} from n distributed nodes, where each data stream di contains a timestamp t and a feature vector x; calculate the temporal adjacency matrix A based on the timestamp difference |ti-tj| and the feature vector difference ||xi-xj|| of adjacent data points in the original data stream set; calculate the feature spectrum λ = {λ1, λ2, ..., λk} and the feature function φi based on the temporal adjacency matrix A, and construct a manifold embedding map f(x); map the original data stream set based on the manifold embedding map f(x) to obtain a unified topological feature representation T = {T1, T2, ..., Tn};

[0024] S12, based on the topological feature representation T and the pre-stored historical feature library H, calculate the cosine similarity of each feature vector to obtain the feature stability matrix S; perform softmax operation on the feature stability matrix S after calculating the row mean to obtain the stability weight w; perform weighted combination of the topological feature representation T, the stability weight w and the mean of the historical feature library to obtain the stabilized feature representation T';

[0025] S13, based on the stabilized feature representation T', calculate the local probability distribution P(x) of the feature vector; based on the local probability distribution P(x), use the Euclidean distance and KL divergence to construct the distance-preserving metric d(x, y), and calculate the distance-preserving error; initialize the compression mapping parameters, and perform gradient descent optimization on the compression mapping parameters by minimizing the distance-preserving error to obtain the optimized parameters; based on the optimized parameters, perform compression mapping on the stabilized feature representation T' to obtain the compressed feature representation C;

[0026] S14. Calculate the mutual information and feature entropy between the compressed feature representation C and the topological feature representation T to obtain the information retention rate I; based on the compressed feature representation and the topological feature representation, calculate the distance matrix of the data points in the compressed feature space and the original feature space, and based on the distance matrix, obtain the topology preservation degree τ by preset rank correlation coefficient; perform weighted combination of the information preservation rate I and the topology preservation degree τ to generate a quality score q and optimization suggestion r.

[0027] In one embodiment of the present application, the distributed data stream dynamic preprocessing is used to achieve efficient preprocessing and feature extraction of multi-source heterogeneous data, providing high-quality input data for subsequent reinforcement learning training. Specifically, the original data stream D = {d1, d2, ..., dn} from n distributed nodes is obtained, each data stream di contains a timestamp t and a feature vector x, and a temporal adjacency matrix A is constructed by a hierarchical topological manifold embedding method. ij =exp(-||xi-xj|| 2 / σ 2)·exp(-|ti-tj| / τ*), where exp represents the exponential function; calculate the topological feature spectrum λ={λ1,λ2,...,λk}, where k is the adaptively determined feature dimension, and generate the manifold embedding mapping function f(x)=Σ i (λiφi(x)), where φi is the characteristic function and the unified topological feature representation T={T1, T2, ..., Tn}.

[0028] Obtain the topological feature representation T and the historical feature library H, and construct the feature stability matrix S, S through the spectral stability preservation algorithm ij =cos(Ti, Hj), calculate the stability weight w=softmax(mean(S, axis=1)), where mean represents the average function and axis represents the dimension for calculating the average value; apply dynamic smoothing: T'=w·T+(1-w)·mean(H), and output the stabilized feature representation T'.

[0029] Obtain the stabilized feature T', and define the distance-preserving metric through the distance-preserving compression mapping algorithm: d(x, y)=||xy||2+α·KL(P(x)||P(y)), where P(x) is the local probability distribution of x and α is the adaptive weight; construct the compression mapping: g(x)=Wx+b, so that |d(g(x), g(y))-d(x, y)|<ε, optimize W and b through gradient descent, and output the compressed feature representation C.

[0030] Get the compressed features C and the original features T, and calculate the information retention rate through the multi-dimensional feature quality evaluation method: I=mutual info (C, T) / entropy(T), where mutual info Represents mutual information, entropy represents entropy; evaluates topology preservation: τ=rank correlation (distance matrix (C), distance matrix (T)), where rank correlation represents the rank correlation coefficient, distance matrix Represents the distance matrix; generates the quality score: q=β1I+β2τ, and outputs the feature quality score q and optimization suggestion r.

[0031] Where n represents the number of distributed nodes, t represents the timestamp, x represents the original feature vector, σ and τ* represent the Gaussian kernel parameters and time decay parameters, λ represents the feature spectrum, φ represents the feature function, α represents the KL divergence weight, ε represents the distance-preserving error threshold, and β1 and β2 represent the quality assessment weights.

[0032] This embodiment obtains the original data stream from multiple distributed nodes, and constructs a temporal adjacency matrix based on the difference between the timestamp and the feature vector, which can effectively capture the temporal relationship and feature similarity between the data streams. It not only improves the efficiency of data preprocessing, but also provides high-quality input data for subsequent feature extraction and analysis. By calculating the feature spectrum and feature function, constructing a manifold embedding mapping, mapping the original data stream to a unified topological feature representation, the global and local structural features of the data can be retained, making the subsequent feature analysis and processing more accurate and effective. Based on the topological feature representation and the historical feature library, the feature stability matrix is ​​calculated, and the stability weight is obtained by softmax operation, and dynamic smoothing is performed, which can improve the stability of the feature representation, reduce the influence of noise and outliers, and thus obtain a more reliable feature representation. By calculating the local probability distribution and the distance-preserving metric, optimizing the compression mapping parameters, and obtaining the compressed feature representation, it is possible to reduce the feature dimension while maintaining the original feature information, and improve the compactness and computational efficiency of the feature representation. By calculating the mutual information and feature entropy between the compressed feature representation and the topological feature representation, evaluating the information retention rate and the topological retention degree, generating quality scores and optimization suggestions, the quality of the feature representation can be comprehensively evaluated, and a scientific basis for model optimization can be provided. This embodiment achieves efficient data preprocessing, stable feature representation, compact feature compression and comprehensive feature quality assessment through multi-level and multi-dimensional feature processing and optimization methods, providing a solid foundation for subsequent reinforcement learning training and model optimization.

[0033] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:

[0034] S21. Based on the compressed feature representation C and the quality score q, a state manifold M is constructed in the compressed feature space, and the tangent space Tx(M) at each state point is calculated; based on the tangent space Tx(M), the gradient ▽xf of the preset value function f is calculated for each state point, and the Riemann metric tensor g is constructed; based on the Riemann metric tensor g, the geodesic equation ▽ γ' γ'=0 generates the current trajectory γ(t) and obtains the geodesic distance d g ; Based on the geodesic distance d g and quality score q, calculate the structured reward function r(s, a), and generate the state-action mapping relationship;

[0035] S22. Obtain the node strategy set π={π1, π2, ..., πn} of the distributed nodes, calculate the Wasserstein metric W(π1, π2) between any two strategies on the strategy manifold based on the structured reward function r(s, a) and the node strategy set π={π1, π2, ..., πn}, and simultaneously calculate the policy gradient ▽πiJ of each strategy πi, including the Q value and the expected term of the state value function V; combine the policy gradient and the Wasserstein metric for collaborative update to obtain the optimized strategy set π' and the strategy consistency metric c;

[0036] S23. Based on the optimization strategy set π' and the pre-stored historical experience pool E, calculate the state-action distribution entropy H(τ) of each experience trajectory τ; based on the state-action distribution entropy H(τ), calculate the importance weight w(τ); combine the importance weight w(τ) and the structured reward function r(s, a) to build a priority queue Q; select top-k samples from the priority queue Q and randomly sample samples from the historical experience pool E to form an updated experience pool E';

[0037] S24. Based on the updated experience pool E', collect the policy parameters θi and gradient ▽θi of each distributed node; based on the policy parameters θi and gradient ▽θi, construct an exponential mapping exp from the tangent space to the manifold p (v) Calculate the Karcher mean θ* that minimizes the sum of squared geodesic distances between parameters; Calculate the logarithmic mapping log of the Karcher mean θ *(θi); based on logarithmic mapping log θ *(θi) and the pre-stored historical momentum m, calculate the new momentum; map the new momentum back to the parameter manifold through exponential mapping to obtain the synchronized global parameter θ new and the consistency index κ.

[0038] In one embodiment of the present application, through distributed strategy optimization and collaborative learning, efficient collaborative learning and strategy optimization among multiple nodes are realized to ensure the stability and convergence of the learning process. Specifically, the compressed features C and the quality score q are obtained, and the tangent space Tx(M) on the state manifold M is constructed through the differential geometry reward construction algorithm, and the Riemann metric tensor g is defined: g(X, Y) = 〈▽xf, ▽γf〉, where f is the value function, ▽ represents the differential operator / gradient, and γ represents the geodesic or trajectory; the geodesic γ(t) is calculated: ▽ γ' γ'=0, generating structured rewards: r(s, a)=-d g (γ(t), γ*(t))+η·q, where d g is the geodesic distance, γ* is the optimal trajectory, η represents the quality score weight, and outputs the structured reward function r and the state-action mapping relationship map.

[0039] Get the reward function r and the node strategy set π = {π1, π2, ..., πn}, and construct the Wasserstein metric on the strategy manifold P through the strategy manifold consistency optimization method: W(π1, π2) = inf γ ∫∫c(x,y)dγ(x,y), define the policy gradient operator: ▽πJ=E[∑(▽logπ(a|s)·(Q(s,a)-V(s)))], apply the collaborative update rule: π'i=πi+α·(▽πiJ+β∑ j W(πi, πj)), where α, β represent the learning rate and consistency weight, and output the optimized strategy set π' and consistency measure c.

[0040] Obtain the optimization strategy π' and the historical experience pool E, calculate the experience trajectory entropy through the dynamic entropy weight experience screening algorithm: H(τ)=-∑p(s,a)logp(s,a), define the importance weight: w(τ)=exp(λ·(H(τ)–H*)), where H* represents the average value of the experience trajectory entropy and λ represents the entropy weight coefficient; build the priority queue Q: rank(τ)=w(τ)·r(τ), and dynamically update the experience pool: E'=top k Q∪random sample (E), output the updated experience pool E' and experience weight w.

[0041] Obtain the strategy parameters θi and gradient ▽θi of each node from the updated experience pool E', and construct an exponential mapping on the parameter manifold M through a nonlinear synchronous aggregation algorithm: exp p (v) = γ v (1), γ v (0)=p,γ v' (0)=v; calculate the Karcher mean: θ*= argmin θ ∑d M (θ,θi) 2 , where argmin represents the independent variable corresponding to the minimum value; applying momentum correction: m=μ·m+(1-μ)·log θ *(θi) θ new =exp θ *(m), where μ represents the momentum coefficient, θ represents the strategy parameter, and d M Represents the distance metric on the manifold; outputs the synchronized global parameter θ new and the consistency index κ.

[0042] This embodiment can accurately capture the geometric structure of the feature space by constructing a state manifold in the compressed feature space and calculating the tangent space and the Riemann metric tensor, which not only improves the accuracy of feature representation, but also provides a solid foundation for subsequent value function calculation and trajectory generation; the current trajectory is generated based on the geodesic equation and the geodesic distance is calculated, which can effectively reflect the real distance between state points. The structured reward function is calculated in combination with the quality score, making the reward function more reasonable and effective, and improving the effect of policy optimization; by calculating the Wasserstein metric and policy gradient on the policy manifold and performing collaborative updates, the policy set can be effectively optimized. The introduction of the policy consistency metric ensures the consistency of the policies of each node and improves the coordination and stability of the overall policy; based on the state-action distribution entropy and the importance weight, a priority queue is constructed, and samples are selected from it to update the experience pool, which can ensure that important experiences are given priority and improve the quality and update efficiency of the experience pool; by calculating the Karcher mean and logarithmic mapping, and combining the historical momentum for momentum correction, the global parameters can be effectively synchronized. The application of exponential mapping ensures the stability and consistency of parameter updates and improves the representativeness of global parameters. This embodiment achieves efficient feature representation, accurate trajectory generation, reasonable reward function construction, effective strategy optimization and stable global parameter synchronization through multi-level feature processing, trajectory generation, strategy optimization and parameter synchronization methods, which provides a solid foundation for the training and optimization of reinforcement learning models and improves the performance and stability of the model.

[0043] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:

[0044] S31, based on the synchronization global parameter θ new and consistency index κ, divide the parameter space into k abstract levels L={L1, L2, ..., Lk}, construct a reversible mapping function ψij between adjacent levels; based on the reversible mapping function ψij, calculate the Frobenius norm of the mapping combination and the unit mapping, and obtain the structural stability index s(L); construct the Hessian matrix of the hierarchical structure, and calculate the trace of the Hessian matrix; based on the structural stability index and the trace of the Hessian matrix, generate the optimized parameter structure L'; where k is a preset natural number;

[0045] S32. Based on the optimized parameter structure L' and the pre-stored historical model set M*, a knowledge graph G is constructed, where the nodes of the knowledge graph represent knowledge units, and the edges represent knowledge associations. The difference between new and old predictions is calculated at the same time. The PageRank value of each knowledge unit in the knowledge graph is calculated and the importance score I(v) is obtained in combination with its timeliness. Based on the importance score and the difference between new and old predictions, the knowledge retention loss L is calculated. keep; Based on the knowledge retention loss, selective knowledge update is performed to obtain the updated knowledge base K';

[0046] S33, obtain the current environment feature E, based on the updated knowledge base K', map the current environment feature to the feature manifold M through the feature mapping function φ E ; Based on the feature manifold, the invariance measure D(e1, e2) between environmental features is calculated in the reproducing kernel Hilbert space; Based on the invariance measure D(e1, e2), the environmental variation V(e) is calculated for the standard environment e; Based on the environmental variation and the regularization term of the policy gradient, the generalization strategy π* is generated;

[0047] S34. Based on the generalization strategy π* and the preset system deployment constraint C, the gradient outer product of the loss function on the parameter is calculated to obtain the parameter sensitivity matrix S; based on the parameter sensitivity matrix S, the importance weight value w(θ) of each parameter is calculated; based on the importance weight value w(θ), structured pruning is performed to obtain the pruned parameter θ pruned ; Perform b-bit quantization on the pruned parameters to obtain the final compression model parameters θ final , while ensuring that the performance loss ΔL is less than the threshold ε.

[0048] In one embodiment of the present application, dynamic model optimization and generalization enhancement are used to improve the robustness and generalization ability of the model, while ensuring the knowledge retention and migration efficiency during the training process. Specifically, the global parameter θ is obtained. new And consistency index κ, through the manifold segmentation optimization algorithm, construct the parameter manifold hierarchical structure: L = {L1, L2, ..., Lk}, each layer Li represents a different level of abstraction, define the inter-layer mapping function: ψij:Li→Lj, satisfying the reversibility condition; calculate the structural stability index: s(L) = ∑ ij ||ψij⊕ψji-Id|| F / (k(k-1)), where ⊕ represents the function composition operator, ψij represents the mapping function from layer Li to layer Lj, ψji represents the mapping function from layer Lj to layer Li, and Id is the unit mapping; optimize the inter-layer connection: L'=argmin L (s(L)+ω·tr(H L )), where H L is the Hessian matrix of the hierarchical structure, ω represents the structural optimization weight, tr represents the matrix trace, and the output is the optimized parameter structure L' and the structural stability index s.

[0049] Obtain the optimized parameter structure L' and historical model set M, and construct the knowledge graph G = (V, E) through the dynamic knowledge fusion algorithm, where the node v∈V represents the knowledge unit and the edge e∈E represents the knowledge association; calculate the knowledge importance score: I(v) = PageRank(v) · exp(-ρ · age(v)), where ρ represents the knowledge decay rate, and define the knowledge retention loss: L keep =∑ v∈V I(v)·||f new (v)-f old (v)|| 2 , where f new (v) represents the feature representation of node v in the current model, f old (v) represents the feature representation of node v in the historical model; selective knowledge update is applied: K' = argmin K (L keep +ξ·R(K)), where R(K) is the knowledge complexity regularization term, ξ represents the complexity weight, and outputs the updated knowledge base K' and knowledge retention rate p.

[0050] Obtain the knowledge base K' and the current environment features E, and construct the environment feature manifold through the environment invariance learning algorithm: M E ={(e, φ(e))|e∈E}, where φ is the feature mapping function; calculate the invariance measure: D(e1, e2)=||φ(e1)-φ(e2)|| H , where H is the RKHS space; generating environmental variation: V(e)=▽ e D(e, e*), where e* is the standard environment, ▽ is the gradient operator, and the optimization strategy generalization: π*=argmin π E e [L(π,e)+χ||▽ e π||2], where χ represents the generalization regularization coefficient, and outputs the generalization enhancement strategy π* and the environmental fitness a.

[0051] Get the generalization strategy π* and deployment constraint C, simplify the algorithm through the adaptive model, and construct the parameter sensitivity matrix: S=ΨL / ΨθΘΨL / Ψθ T , where Ψ is the partial derivative and Θ is the tensor product, and the importance of the parameters is calculated: w(θ)=||S θ || F / sqrt(tr(S θ T S θ ), apply structured pruning: θ pruned =mask(θ, w>τ), where τ is the adaptive threshold; perform quantization optimization: θ final =quantize(θ pruned, bits=b), keep the performance loss ΔL<ε, where b is the number of quantization bits and ε is the performance loss threshold; output the compressed model parameters θ final and performance index m.

[0052] This embodiment can effectively capture the hierarchical structure of the parameter space by dividing the parameter space into multiple abstract levels and constructing a reversible mapping function between adjacent levels; calculate the Frobenius norm of the mapping combination and the unit mapping to obtain a structural stability index, which is helpful to evaluate and optimize the stability of the parameter structure; optimize in combination with the trace of the Hessian matrix to ensure that the parameter structure remains stable and consistent during the transmission process between different levels; construct a knowledge graph based on the optimized parameter structure and the historical model set, which can systematically represent and manage knowledge units and their associations; obtain a knowledge importance score by calculating the PageRank value and combining it with timeliness, ensuring that important knowledge units are retained and updated first; calculate the difference between new and old predictions, evaluate the knowledge retention loss, and perform selective knowledge update New, can effectively maintain and transfer knowledge, improve the robustness and generalization ability of the model; map the current environmental features to the feature manifold through the feature mapping function, and calculate the invariance measure between environmental features in the reproducing kernel Hilbert space, which can accurately evaluate the changes in environmental features; calculate the environmental variation based on the invariance measure, generate a generalization strategy, and ensure the stability and adaptability of the model in different environments; based on the generalization strategy and system deployment constraints, calculate the gradient outer product of the loss function to the parameters to obtain the parameter sensitivity matrix; calculate the importance weight value of each parameter through the parameter sensitivity matrix, and perform structured pruning, which can effectively reduce the model parameters and improve the computational efficiency and deployment performance of the model; perform quantization on the pruned parameters to ensure that the performance loss is controlled within an acceptable range while compressing the model parameters. This embodiment achieves the improvement of the robustness and generalization ability of the model through multi-level parameter optimization, knowledge retention, environmental feature mapping and parameter sensitivity analysis, while ensuring the knowledge retention and migration efficiency during the training process, providing a solid foundation for the stability and efficiency of the model in practical applications.

[0053] like Figure 5 As shown, according to one aspect of the present application, step S4 is further:

[0054] S41, obtain the performance index of the system, based on the final compression model parameter θ final and performance indicators, collect the resource state vector R of each node v =(cpu v , mem v ,net v ); Based on the resource state vector, the computational complexity c of the final compression model parameters at each node is calculated v =complexity(θfinal , R v ), where complexity represents the complexity function; based on the computational complexity, a load distribution matrix is ​​constructed to generate an initial parameter distribution scheme D; where cpu v Indicates the CPU resource status of node v, mem v Indicates the memory resource status of node v, net v Indicates the network resource status of node v;

[0055] S42, based on the initial parameter distribution scheme D, calculate the communication cost between nodes comm(i, j)=size(θ ij )·latency(i, j); Based on the communication cost, construct the node affinity matrix A; Based on the node affinity matrix, iteratively optimize the initial parameter distribution scheme D D {∑ v c v +λ·∑ ij comm(i, j)}, generates the optimized distribution scheme D'; where size represents the parameter size, latency represents the communication delay between nodes, and c v represents the computational complexity of node v;

[0056] S43. Based on the optimized distribution scheme D', calculate the model execution performance p of each node v =throughput(θ v , R v ); Based on the model execution performance, build the task scheduling matrix S; Based on the task scheduling matrix, build the load balancing objective function balance(S)=max v {p v} / avg v {p v}, generate the task scheduling strategy S*; where throughput represents the throughput, θ v Represents the model parameters of node v, avg v represents the average value of all nodes v;

[0057] S44, based on the task scheduling strategy S*, perform distributed model synchronization sync(θ v ,θ global ), get the synchronization result; based on the synchronization result, calculate the synchronization overhead and gradient aggregation delay t sync ; Based on the gradient aggregation delay, optimize the synchronization period T=optimize(t sync , p v ), generate the final deployment configuration C deploy . Where θ globalRepresents the model parameters after global synchronization, and optimize represents the function of optimizing the synchronization cycle.

[0058] In one embodiment of the present application, the compression model parameter θ is obtained. final , calculate the computational complexity of each layer parameter o l =ops(θ1); construct resource consumption vector r for each computing operator op =(flops, memory, bandwidth); Calculate the operator dependency graph G based on the resource vector op ; Generate model to calculate feature matrix M comp .

[0059] Get the calculated feature matrix M comp , collect the hardware specifications spec of each node v v =(gpu type , memory size , network card );Measure node computing power bench v =(float perf , mem bandwidth ,pcie speed );Calculate the node resource score s v =score(spec v , bench v );Generate node performance profile P node .

[0060] Get the node performance profile P node , construct the hardware affinity matrix H ij =affinity(M compi , P nodej ); Calculate the cross-node communication cost comm ij =latency(i, j)·bandwidth(i, j); construct the deployment cost matrix C based on affinity and communication overhead deploy ; Generate preliminary allocation plan A init .

[0061] Get the preliminary allocation plan A init , calculate the load balancing index b of each node v =load(v) / capacity(v); construct the load migration cost function cost(i, j)=move time (i, j)+sync overhead (i, j); optimize the load balancing target min L {std(b v)+λ·sum(cost)}, where λ represents the balancing factor; and generates a load distribution matrix L.

[0062] Get the load distribution matrix L and construct the parameter sharding strategy shard(θ)={θ1,...,θ k}; Calculate the data dependency strength d between shards ij =dependency(θ i ,θ j ); Optimize the placement of shards based on dependency strength place(θ i )=argmin v {comm cost (v)}; generate the final parameter distribution scheme D. Where flops represents the computing power of the operator, memory represents the memory resources required by the operator, bandwidth represents the data transmission rate required by the operator, and gpu type Indicates GPU type, memory size Indicates memory size, network card Indicates the network interface specifications of the node, float perf Indicates floating point performance, mem bandwidth Indicates memory bandwidth, pcie speed Indicates the data transmission rate of the PCIe interface of the node, affinity indicates affinity, latency indicates delay, load indicates load, capacity indicates capacity, move indicates time Indicates the migration time, sync overhead represents synchronization overhead, std represents standard deviation, and dependency represents dependency.

[0063] This embodiment can effectively distribute computing tasks and ensure that the resources of each node are fully utilized by collecting the resource state vector (CPU, memory, network) of each node and calculating the computational complexity of each node based on these resource states; constructing a load distribution matrix and generating an initial parameter distribution scheme helps to balance the load of each node and avoid resource waste and performance bottlenecks. Based on the initial parameter distribution scheme, the communication cost between nodes is calculated, and the node affinity matrix is ​​constructed; through iterative optimization, the communication overhead between nodes can be minimized and the overall efficiency of the system can be improved; the optimization of communication cost is particularly important for distributed systems because it directly affects the response speed and throughput of the system. Based on the optimized distribution scheme, the model execution performance of each node is calculated, and a task scheduling matrix is ​​constructed; through the load balancing objective function, a task scheduling strategy is generated to ensure that the workload of each node is evenly distributed and improve the stability and reliability of the system; load balancing can prevent some nodes from being overloaded, thereby improving the overall performance of the system. Based on the task scheduling strategy, the distributed model synchronization is performed, the synchronization overhead and gradient aggregation delay are calculated, and the synchronization cycle is optimized; by optimizing the synchronization cycle, the synchronization overhead can be reduced, the system training efficiency and model update speed can be improved, among which synchronization optimization is crucial for distributed training because it directly affects the convergence speed and final performance of the model. The final deployment configuration is generated through the above steps to ensure that the system can work efficiently and stably in actual operation. The final deployment configuration not only considers the computational complexity and communication cost, but also integrates the execution performance and load balancing of the nodes to ensure that the system can maintain efficient operation in various environments. This embodiment achieves a comprehensive improvement in system performance through multi-level resource optimization, communication cost minimization, task scheduling and load balancing, distributed synchronization and optimization, and provides a solid foundation for the efficient operation of distributed systems, ensuring efficiency and stability during model training and reasoning.

[0064] According to one aspect of the present application, step S11 is further:

[0065] S111. Obtain the original data stream set D={d1, d2, ..., dn} from n distributed nodes, extract the timestamp sequence Ti={t1, t2, ..., tm} and the feature sequence Xi={x1, x2, ..., xm} in each data stream di; calculate the statistical moments μi and Σi of the feature sequence and perform standardized transformation to obtain the standardized features X'i; use a sliding window of a predetermined length to perform time series segmentation on the standardized features to generate a feature segment set Si; calculate the statistical feature vector Fi of the feature segment set Si.

[0066] S112, based on the feature segment set Si, calculate the autocorrelation matrix Ri = X'i·X'i of each feature segment T; Perform eigenvalue decomposition on the autocorrelation matrix Ri to obtain the eigenvalue sequence λi; Based on the size of the eigenvalue sequence, select the first k eigenvectors to construct the projection matrix Pi; Apply the projection matrix to the feature fragment set to perform dimensionality reduction transformation to obtain the low-dimensional representation Li;

[0067] S113. Based on the low-dimensional representation Li, calculate the Euclidean distance matrix Dij=||Li-Lj||2 between different data points; based on the Euclidean distance matrix, determine the k nearest neighbor points of each data point and construct the adjacency matrix A; based on the adjacency matrix A, calculate the weight matrix Wij=exp(-Dij 2 / σ 2 ); Based on the weight matrix, the graph Laplacian matrix L=DW is calculated, where D is the degree matrix;

[0068] S114. Based on the graph Laplace matrix and the timestamp sequence, construct a timing constraint matrix C, Cij=exp(-|ti-tj| / τ); combine the graph Laplace matrix and the timing constraint matrix to construct an objective function Q(v)=v T Lv+αv T Cv; Based on the objective function, solve the generalized eigenvalue problem Q(v)v=λv to obtain the global coordinate matrix V; Based on the global coordinate matrix V, construct the manifold mapping function f(x);

[0069] S115. Apply the manifold mapping function to all data points to obtain the initial topological feature representation; based on the initial topological feature representation T, calculate the mutual information matrix Iij=MI(Ti, Tj) between different node feature representations; based on the mutual information matrix, construct the consistency loss function Lcons=∑ij(1-Iij) 2 ; The consistency loss function Lcons is minimized by the gradient descent method to obtain a unified topological feature representation T*.

[0070] This embodiment obtains the original data stream from the distributed nodes, extracts the timestamp and feature sequence, calculates the statistical moment and performs the standardized transformation, so as to effectively process and normalize the multi-source heterogeneous data; adopts the sliding window to segment the time series, generates the feature segment set, and calculates the statistical feature vector to ensure the comprehensiveness and accuracy of the feature extraction; calculates the autocorrelation matrix based on the feature segment set, performs the eigenvalue decomposition, selects the first k eigenvectors to construct the projection matrix, and performs the dimensionality reduction transformation, so as to reduce the data dimension while retaining the main information, improve the calculation efficiency and the compactness of the feature representation; by calculating the Euclidean distance matrix, determining the nearest neighbor point, constructing the adjacency matrix and the weight matrix, and further calculating the graph Laplacian matrix, it is possible to effectively capture the relationship and structural characteristics between the data points, and provide a basis for the subsequent manifold learning and feature mapping; combined with the time The time-stamp sequence constructs a timing constraint matrix, and is combined with the graph Laplacian matrix to construct an objective function. By solving the generalized eigenvalue problem, the global coordinate matrix and manifold mapping function are obtained, which can simultaneously consider the spatial and temporal characteristics of the data and improve the accuracy and consistency of the feature representation; the mutual information matrix is ​​calculated based on the initial topological feature representation, and the consistency loss function is constructed. After optimization through the gradient descent method, a unified topological feature representation is obtained, which can ensure the consistency of feature representations of different nodes, reduce the influence of noise and outliers, and improve the stability and reliability of feature representation; this embodiment realizes efficient data preprocessing and feature representation optimization through multi-level feature extraction, dimensionality reduction, adjacency matrix construction, timing constraints and consistency optimization, provides high-quality input data for subsequent model training and optimization, and improves the robustness and generalization ability of the model.

[0071] According to one aspect of the present application, step S22 is further:

[0072] S221. Obtain the node strategy set π={π1, π2, ..., πn} of the distributed nodes, sample the trajectory τi={s1, a1, r1, ..., st, at, rt} from the node strategy set, and calculate the cumulative reward R(τi)=∑ t γ' t r t ; Based on the cumulative reward, calculate the advantage function A(s, a) = Q(s, a) - V(s) of each state-action pair (s, a); Based on the advantage function, calculate the policy gradient ▽πiJ = E[▽logπi(a|s)·A(s, a)]; where γ' t represents the t-th power of the discount factor γ', which is used to discount future rewards when calculating cumulative rewards, and E[⋅] represents the expected value function;

[0073] S222. Based on the policy gradient ▽πiJ, construct the geodesic γ(t) on the policy distribution space P, and calculate the Wasserstein distance W(π1,π2)=inf between any two policy distributions π1 and π2. γ ∫∫c(x, y)dγ(x, y); Based on geodesics, calculate the exponential mapping exp of the policy distribution π (v) and the logarithmic mapping log π (π'), generating a strategy transition path T(π1, π2); where inf represents the infimum, c(x, y) represents the cost function, and dγ(x, y) represents the differential element of the joint distribution used in the Wasserstein distance calculation;

[0074] S223. Based on the strategy transfer path T(π1, π2) and the Wasserstein distance W, calculate the distance matrix D for each pair of strategies π ; Based on the distance matrix, construct the strategy neighbor graph G π ; Calculation strategy The centrality index c(πi)=∑ j exp(-W(πi,πj)), assigns a synergy weight w(πi) to each strategy based on the centrality index;

[0075] S224. Calculate the weighted average gradient ▽π based on the collaborative weight w(πi) and the policy gradient ▽πiJ avg =∑ i w(πi)·▽πiJ; Update each policy parameter θi=θi+α·▽π based on the weighted average gradient and policy transfer path avg , get the optimization strategy set π'; based on the optimization strategy set, calculate the KL divergence matrix KL(πi||πj);

[0076] S225. Based on the optimization strategy set π' and the KL divergence matrix, calculate the strategy consistency index C = exp(-mean(KL)); based on the strategy consistency index, calculate the learning rate α' = α·C; based on the learning rate, calculate each strategy π i 'Performance index J(π i '), generate the final optimization strategy set π* and consistency measure c.

[0077] This embodiment can effectively evaluate the performance of each strategy by sampling trajectories from the strategy set of distributed nodes and calculating the cumulative reward of each trajectory based on the structured reward function, calculate the advantage function of the state-action pair, and calculate the policy gradient based on this to ensure the directionality and effectiveness of the strategy optimization process. Based on the policy gradient, the geodesic in the strategy distribution space is constructed, and the Wasserstein distance between any two strategy distributions is calculated, which can accurately measure the difference between the strategy distributions and ensure a smooth transition in the strategy optimization process; by calculating the exponential mapping and logarithmic mapping of the strategy distribution, the strategy transfer path is generated to further optimize the strategy update process. Based on the strategy transfer path and Wasserstein distance, the distance matrix of each pair of strategies is calculated, and a strategy neighbor graph is constructed. By calculating the centrality index on the strategy neighbor graph, a synergistic weight is assigned to each strategy to ensure the synergy of each strategy in the strategy optimization process, which can improve the overall effect of strategy optimization and ensure the coordination and consistency between strategies. Based on the synergistic weight and strategy gradient, the weighted average gradient is calculated, and each strategy parameter is updated based on this, which can effectively integrate the optimization information of each strategy, improve the efficiency and effect of strategy update, calculate the KL divergence matrix of the optimized strategy set, and further evaluate the consistency between strategies. Based on the optimized strategy set and the KL divergence matrix, the strategy consistency index is calculated, and the learning rate is adjusted based on this. Through this method, the learning rate can be dynamically adjusted to ensure the stability and convergence of the strategy optimization process. The optimized strategy set and consistency metric finally generated can effectively improve the overall performance and consistency of the strategy. This embodiment achieves high efficiency and stability of the policy optimization process through multi-level policy optimization, Wasserstein distance calculation, policy neighbor graph construction and consistency evaluation, provides a solid foundation for policy optimization in distributed systems, ensures the coordination and consistency of policies among different nodes, and improves the overall performance and robustness of the system.

[0078] According to one aspect of the present application, step S31 is further:

[0079] S311, based on the synchronization global parameter θ new and consistency index κ, divide the parameter space into k subspaces {θ1, θ2, ..., θk}; calculate the parameter gradient gi = ΨL / Ψθ in each subspace i ; Based on the parameter gradient, construct the parameter correlation matrix R, where Rij=cos(gi,gj); perform spectral clustering on the parameter correlation matrix to obtain the initial hierarchical structure L={L1, L2, ..., Lk};

[0080] S312. Based on the initial hierarchical structure L, calculate the mapping matrix Mij between adjacent levels so that Lj=Mij·Li; calculate the condition number κ(Mij)=||Mij||·||Mij for each mapping matrix-1 ||; Based on the condition number, construct the inter-layer stability index sij=exp(-κ(Mij)) and generate the inter-layer connection strength matrix S;

[0081] S313, based on the inter-layer connection strength matrix S, calculate the Hessian matrix Hi = Ψ of each layer 2 L / Ψθi 2 ; Calculate the eigendecomposition of the Hessian matrix to obtain the eigenvalue λi and eigenvector vi; Based on the eigenvalue and eigenvector, calculate the intra-layer complexity ci=tr(Hi) / max(λi) and generate the layer complexity vector C;

[0082] S314, based on the layer complexity vector C and the inter-layer connection strength matrix S, construct the hierarchical optimization objective function Q(L)=∑ ij sij+ω·∑ i ci; calculate the hierarchical structure gradient of the hierarchical optimization objective function ▽ LQ ; Based on the hierarchy gradient, update the hierarchy L'=L-η·▽ LQ , generate the optimized hierarchical structure L'; where ∑ represents the summation operator;

[0083] S315. Based on the optimized hierarchical structure L', calculate the information flow matrix Iij = MI (Li, Lj) between the hierarchies; based on the information flow matrix, construct the structural stability index s (L) = min ij (Iij); Based on the structural stability index, normalize the parameters of each layer to obtain the standardized hierarchical structure L* as the optimization parameter structure. Among them, θ is the model parameter, κ is the consistency index, L is the loss function, g is the parameter gradient, R is the correlation matrix, M is the mapping matrix, H is the Hessian matrix, λ is the eigenvalue, v is the eigenvector, s is the stability index, c is the complexity index, ω is the weight coefficient, η is the learning rate, I is the information flow matrix, MI is the mutual information function, tr is the trace of the matrix, Ψ is the partial derivative operator, and ▽ is the gradient operator.

[0084] This embodiment divides the parameter space into multiple subspaces, calculates the parameter gradient of each subspace, and constructs a parameter correlation matrix, which can effectively capture the relationship between parameters. Spectral clustering is performed on the parameter correlation matrix to obtain an initial hierarchical structure, which helps to identify the hierarchical relationship and structural characteristics of the parameters. The mapping matrix between adjacent levels is calculated based on the initial hierarchical structure, and the stability of the mapping matrix is ​​evaluated by the condition number to construct an inter-layer connection strength matrix. This method can ensure that the hierarchical structure remains stable and consistent during the transfer process between different levels, and improve the robustness and reliability of the model; by calculating the Hessian matrix of each layer and performing eigendecomposition to obtain eigenvalues ​​and eigenvectors, the complexity of each layer parameter can be evaluated, and the complexity within the layer is calculated based on the eigenvalues ​​and eigenvectors to generate layer complexity vectors, which helps to identify and optimize key parameters in the hierarchy; based on the layer complexity vector and the inter-layer connection strength matrix, a hierarchical optimization objective function is constructed, and the hierarchy gradient is calculated and the hierarchy is updated, which can dynamically adjust and optimize the hierarchy to ensure that the model remains efficient and stable during the parameter transfer process between different levels; based on the optimized hierarchy, the information flow matrix between levels is calculated, and a structural stability index is constructed to evaluate the overall stability of the hierarchy. By normalizing the parameters of each layer, a standardized hierarchy is obtained to ensure that the model remains consistent and stable during the parameter transfer process between different levels. This embodiment achieves efficient management and optimization of model parameters through multi-level parameter space division, correlation analysis, inter-layer mapping, Hessian matrix calculation and hierarchical optimization.

[0085] According to one aspect of the present application, it also includes:

[0086] S51, based on the final deployment configuration C deploy and the final compression model parameters θ final , construct the Lyapunov function V(θ), calculate the parameter gradient norm ||▽θ|| and the update step size ||Δθ||; based on the parameter gradient norm and the update step size, construct the safe region Ω={θ:V(θ)≤c}, and generate the safe training constraint K; where c represents the threshold of the Lyapunov function;

[0087] S52. Based on the security training constraint K, calculate the feature importance score (fi) for the model decision sequence {(st, at, rt)}; based on the feature importance, construct the decision explanation graph G exp , extract the key decision path P = {p1, ..., pk}, and generate the decision explanation model M exp ;

[0088] S53, Computational Decision Explanation Model M expBased on the performance indicators perf(θ, env) in each scenario, the evaluation indicator space Ω={accuracy, latency, stability} is constructed to generate multi-dimensional indicators. Based on the multi-dimensional indicators, the comprehensive score E=evaluate(θ, Ω) is calculated to generate a performance evaluation report R. Among them, accuracy represents the accuracy indicator of the model in each scenario, latency represents the delay indicator of the model, and stability represents the stability indicator of the model. Evaluate represents the evaluation function.

[0089] S54. Based on the performance evaluation report R, construct the feedback optimization target J(θ)=E+μ·V(θ), calculate the parameter adjustment direction Δθ=▽J(θ); based on the parameter adjustment direction, use the adaptive learning rate η=adapt(||▽J||) to generate the parameter update suggestion U param And feedback to step S3. Where E represents the evaluation score, μ represents the weight coefficient, and adapt represents the adaptive learning rate function.

[0090] This embodiment can define a safe area by constructing a Lyapunov function and calculating the parameter gradient norm and update step size to ensure stability and security during model training. The generated safe training constraints can effectively prevent instability or divergence during the model parameter update process and improve the reliability of the training process. Based on the safe training constraints, the feature importance of the model decision sequence is calculated, and a decision explanation graph is constructed to extract the key decision path and generate a decision explanation model, which can help understand the decision process of the model, provide transparency and explainability, and facilitate the analysis and optimization of the decision logic of the model. By calculating the performance indicators of the decision explanation model in various scenarios and constructing an evaluation indicator space (including accuracy, latency, and stability), it can fully The performance of the model is evaluated on the surface; a comprehensive score is calculated based on multi-dimensional indicators, and a performance evaluation report is generated to provide a scientific basis for model optimization, which can ensure the efficiency and stability of the model in different application scenarios; based on the performance evaluation report, a feedback optimization target is constructed, the parameter adjustment direction is calculated, and an adaptive learning rate is used for parameter update, which can dynamically adjust the model parameters to ensure the efficiency and convergence of the optimization process; the introduction of an adaptive learning rate can adjust the learning rate according to the gradient norm, thereby improving the flexibility and adaptability of the optimization process; by generating parameter update suggestions and feeding them back to step S3, a closed-loop optimization process is formed to ensure that the model continues to improve during training and deployment, so that the model parameters can be continuously optimized and the overall performance and robustness of the model can be improved. This embodiment achieves the efficiency, stability and interpretability of the model training and optimization process through safe training constraints, feature importance analysis, decision interpretation, multi-dimensional performance evaluation and feedback optimization.

[0091] According to one aspect of the present application, step S52 is further:

[0092] S521, obtain the model decision sequence {(st, at, rt)}, calculate the activation gradient gf=ΨQ / Ψf of each state feature; construct the feature sensitivity matrix S ij =correlation(gi,gj); Perform feature clustering clustering(S) based on the sensitivity matrix; Generate feature group Gf.

[0093] S522, obtain the feature group Gf, calculate the cumulative contribution of each group of features cg = sum(importance(f)); construct the feature time series correlation matrix T ij =temporal corr (gi, gj); extract feature influence link path(g)=extract path (T); Generate feature association graph G feat .

[0094] S523, obtain feature association graph G feat , calculate the decision point attention weight αt=attention(st,at); construct the decision transition probability matrix P ij = transition(ai, sj); Extract key decision sequence key based on attention and transition probability seq ; Generate decision path set P.

[0095] S524, obtain the decision path set P, calculate the path reliability score rp=reliability(p); construct the path coverage matrix C ij =coverage(pi,pj); optimize path selection target max P {sum(rp)-μ·overlap(P)}; Generate the core decision path P*.

[0096] Step S525: Obtain the core decision path P* and construct a lightweight proxy model h(s); train the proxy model to fit the key decision path predict(h)≈P*; calculate the consistency κ=consistency(h,π) between the proxy model and the original decision; generate the final explanation model M exp . Where st represents the state at time t, at represents the action at time t, rt represents the reward at time t, gf represents the activation gradient of feature f, S ij represents the feature sensitivity matrix, μ represents the overlap penalty coefficient, and π represents the original strategy.

[0097] This embodiment can effectively evaluate the correlation between features by calculating the activation gradient of each state feature and constructing a feature sensitivity matrix; perform feature clustering based on the sensitivity matrix to generate feature groups, which helps to identify and group features with similar influence, improve the accuracy of feature analysis, reduce redundant features, and improve the interpretability of the model. Calculate the cumulative contribution of each group of features and construct a feature time series correlation matrix to capture the correlation of features in time series; extract feature influence links and generate feature association graphs to help understand the dynamic relationship between features, reveal the complex interactions between features, and improve the transparency and interpretability of the model; calculate the attention weights of decision points and construct a decision transition probability matrix to identify key decision sequences; extract key decision paths based on attention and transition probabilities and generate decision path sets, which helps to understand the decision process of the model, provide clear decision logic, and facilitate analysis and optimization of the decision strategy of the model; calculate the reliability score of the path and construct a path coverage matrix to evaluate and optimize the choice of decision paths, generate core decision paths by optimizing path selection targets, ensure the reliability and stability of the model at key decision points, improve the decision quality of the model, and reduce the risk of wrong decisions. Constructing a lightweight proxy model and training it to fit the key decision path can simplify the complexity of the model; calculating the consistency between the proxy model and the original decision ensures that the proxy model can accurately reflect the decision logic of the original model, which can improve the interpretability and usability of the model while maintaining the model performance. This embodiment achieves efficient interpretation and optimization of the model through feature sensitivity analysis, time series correlation analysis, decision point attention, path reliability assessment and lightweight proxy model construction, providing a solid foundation for the transparency, interpretability and decision quality of the model, and improving the reliability and usability of the model in practical applications.

[0098] According to one aspect of the present application, step S54 is further:

[0099] S541. Obtain the evaluation report R, extract the performance indicator vector perf = (accuracy, latency, stability); calculate the indicator weight w = importance (perf); construct a multi-objective optimization function f (θ) = weighted sum (w, perf); generate optimization objective function J.

[0100] S542, obtain the optimization objective function J, calculate the parameter gradient ▽J(θ); construct the Hessian matrix H=Ψ²J / Ψθ²; optimize the update direction d=-H based on the curvature information - ¹·▽J; Generate parameter adjustment vector Δθ.

[0101] S543, obtain parameter adjustment vector Δθ, calculate historical update variance var(Δθ); construct adaptive learning rate function η(t)=base lr adapt(var); optimize the momentum factor β = momentum(||▽J||); generate the parameter update strategy U.

[0102] S544, obtain the update strategy U, and construct the parameter constraint space Ω={θ:||θ-θ init ||≤δ}; calculate the projection operator proj Ω (θ); perform constrained optimization to update θ new =proj Ω (θ+η·Δθ); generate new parameters θ new .

[0103] S545, obtain new parameter θ new , calculate the effectiveness score(θ) of parameter update new ); build rollback trigger condition trigger(θ)=early stop (score); Generate the final parameter update suggestion U param . Where base lr represents the basic learning rate, δ represents the constraint radius, and θ init represents the initial parameters, and ||·|| represents the norm.

[0104] This embodiment extracts performance indicators (accuracy, delay, stability) and calculates their weights to construct a multi-objective optimization function, which can comprehensively consider the impact of different performance indicators and ensure that the model performs evenly in all aspects. The generated optimization objective function provides a clear optimization direction for subsequent parameter adjustment. The parameter gradient of the optimization objective function is calculated, and the Hessian matrix is ​​constructed. The update direction is optimized based on the curvature information, which can accurately adjust the parameters and ensure the efficiency and accuracy of the optimization process. The generated parameter adjustment vector provides a specific adjustment plan for the update of model parameters. By calculating the historical update variance, constructing an adaptive learning rate function, and optimizing the momentum factor, it can dynamically adjust the model parameters. The learning rate and momentum are adjusted to ensure the stability and rapid convergence of the parameter update process; the generated parameter update strategy improves the flexibility and adaptability of model training; the parameter constraint space is constructed and the projection operator is calculated, and the constrained optimization update is performed to ensure that the parameter update is carried out within a safe range and avoid instability caused by excessive parameter adjustment; the generated new parameters ensure the stability and security of the model during the optimization process; the effectiveness of the parameter update is calculated, and the rollback trigger condition is constructed to detect and correct bad parameter updates in a timely manner to ensure the continuous improvement of model performance; the generated final parameter update suggestion provides a reliable parameter adjustment scheme to ensure the efficiency and stability of the model in practical applications. This embodiment achieves efficient optimization and stable update of model parameters through multi-objective optimization, parameter gradient calculation, adaptive learning rate, parameter constraints and rollback mechanism. It provides a solid foundation for the training and deployment of the model, ensuring the efficient operation and continuous optimization of the model in practical applications.

[0105] According to one aspect of the present application, it also includes: obtaining the distributed node state set N and the system resource utilization U, building a node topology map and calculating the load metric, performing dynamic load migration to obtain a migration plan M*; building a fault prediction model based on the system state S and historical fault records F, calculating the node criticality and generating a backup strategy; obtaining the communication demand D and network state N between nodes, building a communication traffic tensor and optimizing routing and bandwidth allocation; collecting the system performance indicator set P and the optimization target O, building a performance metric space and performing adaptive parameter tuning. Specifically:

[0106] Obtain the distributed node state set N={n1, n2, ..., nk} and the system resource utilization U; construct the node topology graph G(V, E, W), where V is the node set, E is the communication link set, and W is the performance weight matrix; calculate the CPU utilization cpu of each node v v 、Memory usage mem v and network traffic net v , multiplied by the resource weight vector ζ to obtain the node load metric L(v); calculate the ratio of the maximum node load to the average load to obtain the load imbalance index I imb; Based on the node load measurement and migration cost optimization, the load migration scheme M* is obtained.

[0107] Obtain the system state S and the pre-stored historical fault record F; extract the state features through the state feature extraction function Φ and calculate the fault probability P(f|s); calculate the weighted sum of various types of fault risks risk(v, fi) and impact(fi) for each node v to obtain the node criticality C(v); select the set of nodes that need to be backed up according to the node criticality C(v), calculate the switching time and synchronization overhead of each backup node, and generate the fault recovery strategy R.

[0108] Obtain the communication demand D and network status N between nodes; construct a three-dimensional communication flow tensor T, record the flow flow (i→j, t) from node i to node j at time t; calculate the transmission cost cost (p) and flow (p) for each possible path p, and optimize the routing configuration under the condition of satisfying the link capacity constraint; based on the communication utility function utility (flow ij ) Optimize bandwidth allocation B*; according to the window size between nodes size and queue length queue length Calculate the transmission rate rate(i, j).

[0109] Obtain the system performance indicator set P and optimization target O; construct an n-dimensional performance metric space Ω, which contains all combinations of performance indicators; i (x) and weight α i Multiply and sum to get the optimized objective function f(x); use the simultaneous perturbation stochastic approximation method SPSA to calculate the objective function gradient estimate ▽J*; calculate the control input u(t) based on the performance error e(t), add exploratory perturbations to update the system configuration parameters, and get the optimized configuration x new .

[0110] In one embodiment of the present application, the node state set N = {n1, n2, ..., nk} and the resource utilization U are obtained, and a node topology graph G (V, E, W) is constructed through a dynamic topology balancing algorithm, where V is a node set, E is a communication link, and W is a performance weight matrix; the node load metric is calculated: L (v) = ∑ (cpu v , mem v ,net v )·ζ, where ζ is the resource weight vector; define the load imbalance index: I imb =max v L(v) / avg v L(v); Application dynamic load migration: M*=argmin M ∑ v |L(v)-L- | 2 +δ·cost(M), where cost(M) is the migration cost; output load migration plan M* and balance index B.

[0111] Obtain the system state S and historical fault records F, and build a fault prediction model through the predictive fault-tolerant control algorithm: P(f|s)=softmax(Φ(s) T W), where Φ is the state feature extraction function; calculate the key metric: C(v)=∑ i λ i ·risk(v,f i )·impact(f i ); Generate backup strategy: R={(v, B v )|C(v)>η}, where B v is the backup set of node v; perform automatic recovery: recover(v)=argmin b∈Bv (switch time (b)+sync cost (b)); Output the recovery strategy R and the system reliability index σ.

[0112] Obtain the communication demand D and network status N between nodes, build the communication flow tensor T through the adaptive flow control algorithm: T[i, j, t] = flow(i→j, t), and calculate the optimal route: min∑ p cost(p)·flow(p);stcapacity constraints (p); Dynamic allocation of application bandwidth: B*=argmax B ∑ ij utility(flow ij );st∑ ij B ij ≤B total ; To achieve congestion control: rate(i, j)=min(window size (i, j), queue length (i, j))·μ;

[0113] Output the optimized communication configuration C* and network efficiency E.

[0114] Obtain the system performance indicator set P and optimization target O, and construct the performance measurement space through the multi-objective adaptive optimization algorithm: Ω={(p1, p2, ..., pn)|pi∈P}; define the optimization objective function: f(x)=∑ i α i ·metric i(x), where x is the system configuration vector; apply online parameter tuning: x* =argmin x (f(x)+γ·reg(x)); stconstraints(x); perform adaptive feedback: Δx=-ν·▽f(x)·H(x) -1 , where H is the Hessian matrix of the objective function, and outputs the optimized configuration parameter x* and performance improvement Δp. Where δ is the migration cost weight, λ is the risk weight coefficient, η is the backup threshold, μ is the flow control factor, α is the performance weight vector, γ is the regularization coefficient, and L - is the average load, Φ is the feature extraction function, and H is the Hessian matrix.

[0115] This embodiment can effectively evaluate the load distribution of the system by acquiring the status of distributed nodes and the utilization of system resources, constructing a node topology map, and calculating the node load metric; calculate the load imbalance index and optimize the load migration plan based on the node load metric and migration cost to ensure system load balancing and improve overall performance and resource utilization; extract state features and calculate the probability of failure based on the system status and historical fault records, and predict potential faults in advance. Calculating the failure risk and impact of each node, determining the node criticality, selecting the set of nodes that need to be backed up, and generating a fault recovery strategy can improve the fault tolerance and reliability of the system and reduce the impact of faults on the system; by constructing a three-dimensional communication traffic tensor, recording the traffic between nodes, and calculating the transmission cost and traffic, the routing configuration can be optimized; optimizing bandwidth allocation based on the communication utility function, and calculating the transmission rate to ensure efficient use of communication links can improve the communication efficiency of the system and reduce communication delays and bottlenecks. Obtaining system performance indicators and optimization goals, constructing a performance metric space, and using the simultaneous perturbation random approximation method to calculate the objective function gradient estimate can accurately evaluate and optimize system performance; calculating control inputs based on performance errors, and adding exploratory perturbations to update system configuration parameters to ensure efficient operation of the system in different environments. Through system adaptive tuning and performance assurance, the system's automatic tuning, fault recovery, and performance optimization are achieved to ensure the stability and reliability of the distributed training platform. The system configuration can be dynamically adjusted to enhance the system's adaptive capabilities and stability, ensuring efficient operation of the system in practical applications. This embodiment achieves automatic tuning and performance assurance of the system through load balancing, fault prediction and recovery, communication optimization, performance evaluation and optimization, providing a solid foundation for the stability and reliability of the distributed training platform, and ensuring efficient operation and continuous optimization of the system in practical applications.

[0116] According to one aspect of the present application, generating a fault recovery strategy R specifically includes: obtaining a system state S = {s1, s2, ..., sn} and a pre-stored historical fault record F = {f1, f2, ..., fm}; extracting a time series feature x from the system state t =(cpu t , mem t ,net t , disk t );Calculate the autocorrelation matrix A of the state sequence t =x t ·x t T ; Construct the state eigenvector φ(s) based on the autocorrelation matrix; Generate the enhanced state representation S'.

[0117] Get the step-enhanced state representation S', calculate the prior probability p(fi) of each fault type fi; calculate the conditional probability p(fi|s)=softmax(φ(s) for each state s T W i );Calculate the fault transfer matrix T based on historical fault records ij =p(fj|fi); generate a fault prediction model P(f|s,h), where h is the fault history.

[0118] Obtain the fault prediction model P(f|s, h) and calculate the various fault risk values ​​r for each node v i (v)=p(f i |s v ,h v );Calculate the fault propagation coefficient β based on the system topology ij ; Calculate the fault impact I(f i )=∑ vr _ i (v) β i v; Generate node criticality matrix C.

[0119] Get the step node criticality matrix C and calculate the available resource vector R of each node v =(cpu v , mem v , stor v ); construct a backup cost function cost(v, b) based on resource constraints; select a backup node set Bv that satisfies cost(v, b)<η for each key node v; generate a global backup strategy R.

[0120] Get the backup strategy R, calculate the data synchronization overhead sync(v, b) between nodes; estimate the failover time switch(v, b); construct the recovery time optimization target t(v, b) = switch(v, b) + ε·sync(v, b); generate the final recovery strategy R* and system reliability index σ. t represents the time series feature vector, η represents the cost threshold, ε represents the weight factor, cpu represents CPU usage, mem represents memory usage, net represents network traffic, disk represents disk usage, and stor represents storage capacity.

[0121] This embodiment can capture the dynamic change characteristics of the system state by extracting the time series characteristics of the system state and calculating the autocorrelation matrix; constructing the state feature vector based on the autocorrelation matrix and generating the enhanced state representation is helpful to improve the accuracy and reliability of fault prediction, so that the time series characteristics of the system state can be better reflected and high-quality input data can be provided for subsequent fault prediction. Based on the enhanced state representation, the prior probability and conditional probability of the fault type are calculated, and the fault prediction model is generated, so that the potential fault risk can be identified in advance; by calculating the fault risk value and fault propagation coefficient of each node, evaluating the impact of the fault, and generating the node criticality matrix, the fault risk and criticality of each node in the system can be effectively evaluated, and the fault tolerance and reliability of the system can be improved. Based on the node criticality matrix, the available resource vector of each node is calculated, and the backup cost function is constructed, and the backup node set that meets the cost constraint is selected to generate a global backup strategy, so that resource allocation can be optimized, ensuring that the key nodes can be quickly restored when a fault occurs, and reducing the impact of the fault on the system. The data synchronization overhead and fault switching time between nodes are calculated, the recovery time optimization target is constructed, and the final recovery strategy is generated, which can ensure that when a fault occurs, the system can quickly switch to the backup node, reduce the fault recovery time, and improve the reliability and stability of the system. By calculating the reliability index of the system, the performance of the system under different fault conditions can be comprehensively evaluated; the generated system reliability index provides a scientific basis for the maintenance and optimization of the system, ensuring the efficiency and stability of the system in actual operation. This embodiment realizes the automatic tuning and performance guarantee of the system through time series feature extraction, fault prediction, risk assessment, backup strategy optimization and fault recovery, providing a solid foundation for the stability and reliability of the distributed system, and ensuring the efficient operation and continuous optimization of the system in actual applications.

[0122] According to one aspect of the present application, the method further includes: obtaining the system operation state S and the historical security events H, constructing the security state manifold and calculating the control invariant set, and generating the security strategy π safe; Build a decision explanation diagram based on the model decision sequence D and the environment interaction record E, calculate feature attribution and generate key decision paths; obtain system operation data R and performance indicator set K, build an evaluation metric space and perform multi-dimensional performance analysis; collect evaluation reports R and historical optimization records O, build an optimization objective function and perform closed-loop control optimization. Specifically:

[0123] Obtain the system operation status S and pre-stored historical security events H; construct the security state manifold M based on the Lyapunov function V safe , ensure that the state satisfies the constraint condition V(s)≤c; for each state s, calculate the action set I(s) that keeps the system safe under all possible disturbances d∈D; calculate the current state s to the safety boundary ΨM safe The distance ρ(s) of the security policy is obtained by combining the Q value and the security metric ρ(s) to select the optimal action in the action set I(s) that satisfies the security constraint. safe .

[0124] Obtain the model decision sequence D and the environment interaction record E; construct a decision explanation graph G=(V, E, A), where A represents the attention weight between nodes; calculate the Shapley value of each feature i and other features j and multiply it by the importance weight to obtain the feature attribution score φ(i); find the path P* with the largest attention weight product in the decision explanation graph, while ensuring the connectivity of the path; construct an interpretable model M based on the decision sequence D, the feature attribution score φ and the key path P* exp .

[0125] Obtain system operation data R and performance indicator set K; construct n-dimensional evaluation metric space Ω={ω i}, each dimension corresponds to an evaluation indicator; for each performance indicator k, the evaluation score metric is calculated i (k) and the adaptive weight β i Multiply to obtain a comprehensive performance score S(k); perform a fast Fourier transform on the performance score sequence S(t) and apply the time window function W(t) to obtain the timing analysis result T(t); generate a system evaluation report R based on the performance score, timing analysis results and the preset threshold θ.

[0126] Obtain the evaluation report R and the pre-stored historical optimization record O; construct a multi-objective optimization function J(x) and transform each optimization objective obj i (x) and weight λ i Multiply and sum; use the simultaneous perturbation stochastic approximation method to calculate the gradient estimate ▽J* of the optimization target; calculate the proportional integral control input u(t) based on the performance error e(t); combine the control input u with the exploratory noise to update the system configuration parameters and obtain the optimized configuration x new .

[0127] In one embodiment of the present application, the system operation state S and historical security events H are obtained, and a security state manifold M is constructed by a dynamic security boundary construction algorithm: safe ={s|V(s)≤c}, where V is the Lyapunov function and c is the safety threshold; calculate the control invariant set: I(s)={a|all d∈D, f(s, a, d)∈M safe}, where D is the perturbation set; define the security metric: ρ(s)=dist(s,ΨM safe ), where ΨM safe is the safety boundary; apply safety constraints: π safe (s)=argmax a ∈I(s)[Q(s,a)+ω·ρ(s)]; Output security policy π safe and the security metric ρ.

[0128] Obtain the model decision sequence D and the environment interaction record E, and construct a decision explanation graph through the hierarchical attribution analysis algorithm: G=(V, E, A), where A is the attention weight; calculate the feature attribution score: φ(i)=∑ j Shapley(i, j)·importance(j); Generate critical decision path: P* = argmax P ∏ e∈P A(e), satisfies the path connectivity constraint; builds the explanation model: M exp = distill(D, φ, P*), using an interpretable structure (such as a decision tree), outputs an explanation model M exp and the credibility index τ.

[0129] Obtain system operation data R and performance indicator set K, and construct an evaluation metric space through a multi-dimensional performance evaluation algorithm: Ω={ω i |i∈[1,n]}, contains n evaluation dimensions; calculate performance score: S(k)=∑ i β i ·metric i (k), where β is the adaptive weight vector; perform timing analysis: T(t)=FFT(S(t))·W(t), where W is the time window function; generate an evaluation report: R=aggregate(S, T, threshold=θ*); output the evaluation report R and the improvement suggestion list L.

[0130] Obtain the evaluation report R and historical optimization record O, and construct the optimization objective function through the closed-loop optimization control algorithm: J(x)=∑ i λ i ·obj i(x), where x is the system configuration vector; compute the gradient estimate: ▽J* = SPSA(J, x, δ), using the simultaneous perturbation stochastic approximation; apply adaptive control: u(t) = -K(t) · e(t) - ∫K i ·e(τ)dτ; where e is the performance error; perform parameter update: x new =x+η·u+ξ·noise(), add exploratory disturbances, and output the optimized configuration x new And the optimization effect Δ. Where ω represents the safety weight coefficient, θ* represents the threshold parameter, λ represents the target weight vector, η represents the learning rate, ξ represents the noise coefficient, K represents the control gain matrix, and δ represents the perturbation step size.

[0131] This embodiment ensures that the system state satisfies the safety constraints by constructing a safe state manifold based on the Lyapunov function; calculates the safe action set of each state under all possible disturbances, and selects the optimal action in combination with the Q value and the safety metric to generate a safety strategy, which can effectively prevent the system from entering an unsafe state during operation and improve the stability and safety of the system. By constructing a decision interpretation diagram and calculating the Shapley value of the feature, the contribution of each feature to the decision can be quantified; finding the path with the largest attention weight product, ensuring path connectivity, and generating an interpretable model can provide clear decision logic, help understand the decision-making process of the model, and improve the transparency and interpretability of the model. Based on system operation data and performance indicators, a multidimensional evaluation metric space is constructed, and the comprehensive performance score is calculated; through fast Fourier transform and time window function, time series analysis is performed to generate a system evaluation report, which can comprehensively evaluate the performance of the system in different time periods, provide detailed performance analysis results, and provide a scientific basis for system optimization. By constructing a multi-objective optimization function and calculating the gradient estimate, calculating the proportional integral control input based on the performance error, and updating the system configuration parameters in combination with exploratory noise, the system configuration can be dynamically adjusted to ensure the efficient operation of the system in different environments. The optimized configuration can improve the overall performance and stability of the system. By obtaining evaluation reports and historical optimization records, constructing a multi-objective optimization function, and using the simultaneous perturbation random approximation method for optimization, the continuous improvement of the system configuration can be ensured, and the automatic tuning of the system can be achieved, reducing manual intervention, and improving the system's adaptability and optimization efficiency. This embodiment achieves automatic tuning and performance assurance of the system through security strategies, decision interpretation, performance evaluation, multi-objective optimization, and configuration updates.

[0132] According to one aspect of the present application, an interpretable model M is constructed exp Specifically: obtain the model decision sequence D = {d1, d2, ..., dn} and the environment interaction record E = {e1, e2, ..., em}; extract the state feature s for each decision point di i, action features a i and the reward value r i ; Calculate the temporal correlation strength w between decision points ij =exp(-|t i -t j | / τ'); construct a temporal dependency matrix T based on the correlation strength; generate an enhanced decision sequence D'; where τ' represents the time scale parameter.

[0133] Get the enhanced decision sequence D' and calculate the activation gradient g for each feature dimension k k =Ψa / Ψs k ; Construct attention weight α based on activation gradient k =softmax(||g k ||); Calculate the feature importance vector v for each decision point i =α·s i ; Generate feature attention map A.

[0134] Get the feature attention map A and calculate the mutual information MI(s) for each pair of features (i, j) i ,s j );Build the feature association graph G=(V, E) based on the mutual information between features;Calculate the centrality index c(i)=∑ j MI(s i ,s j ); Generate feature attribution score φ.

[0135] Get the feature attribution score φ and the attention map A, and construct the decision path scoring function score(p)=∏ k∈p α k ; Search for the k decision paths P={p1, p2, ..., pk} with the highest scores; Verify path connectivity and calculate path reliability r(p); Generate a set of critical decision paths P*.

[0136] Obtain the key decision path P* and feature attribution score φ, and construct the interpretability loss function L exp =∑ p (1-r(p)) 2 +λ·∑ i φ i ; Train a lightweight explainer h(s) based on the loss function; Calculate the consistency κ between the explainer prediction and the original decision; Generate the final explanation model M exp And the credibility index τ. Where λ represents the regularization coefficient and ∏ represents the multiplication operator.

[0137] This embodiment can effectively capture the time dependency between decision points by extracting the state features, action features and reward values ​​of each decision point, calculating the temporal correlation strength between decision points, and constructing a temporal dependency matrix; the generated enhanced decision sequence can more accurately reflect the temporal characteristics in the decision process and improve the decision quality and accuracy of the model. Attention weights are constructed based on activation gradients, and feature importance vectors are calculated to generate feature attention graphs, which can quantify the importance of each feature in the decision process, thereby helping to identify key features, improve the interpretability and transparency of the model, and facilitate analysis and optimization of the decision logic of the model. By calculating the mutual information between features, constructing a feature association graph, and calculating the centrality index of each feature node, and generating feature attribution scores, the complex relationships and interactions between features can be revealed, which can provide a deeper feature analysis, help understand the impact of features on decisions, and improve the interpretability of the model. Based on the feature attribution score and the attention graph, a decision path scoring function is constructed, the decision path with the highest score is searched, and the path connectivity and reliability are verified to generate a set of key decision paths, which can identify and evaluate the key paths in the decision process, provide clear decision logic, and improve the transparency and interpretability of the model. By constructing an interpretability loss function and training a lightweight interpreter, calculating the consistency between the interpreter prediction and the original decision, and generating the final interpretation model and credibility index, the complexity of the model can be simplified, and the interpretability and ease of use of the model can be improved. This can provide reliable decision explanations while maintaining model performance, and improve the transparency and user trust of the model. This embodiment achieves efficient interpretation and optimization of the model through time series correlation analysis, feature attention, feature association, decision path scoring, and lightweight interpreter construction. These technical effects provide a basis for the transparency, interpretability, and decision quality of the model, and improve the reliability and usability of the model in practical applications.

[0138] In another embodiment of the present application, a lightweight distributed reinforcement learning training platform based on Ray includes: a data preprocessing module, a distributed learning module, a model optimization module, a system tuning module and a security assessment module.

[0139] Among them, the data preprocessing module is used to perform topological fusion, feature extraction and quality assessment of multi-source data streams. By constructing a temporal adjacency matrix, calculating feature spectra and manifold embedding mapping, the heterogeneous data of distributed nodes are converted into a unified topological feature representation, and the dynamic stability enhancement and adaptive compression of features are achieved.

[0140] The distributed learning module is used to achieve policy optimization and collaborative learning among multiple nodes, construct the reward structure based on differential geometry methods, achieve policy consistency optimization through the Wasserstein metric on the policy manifold, and combine dynamic entropy weights for experience screening and nonlinear parameter synchronization.

[0141] The model optimization module is used to improve the robustness and generalization ability of the model. It constructs a multi-level parameter structure through manifold segmentation optimization, realizes dynamic knowledge fusion and selective knowledge transfer, enhances the generalization of the model based on environmental invariance learning, and performs adaptive model compression.

[0142] The system tuning module is used to achieve dynamic balancing and performance optimization of distributed nodes, ensure system reliability through predictive fault-tolerant control, optimize network resource utilization based on adaptive traffic control, and perform multi-objective performance evaluation and parameter adaptive tuning.

[0143] The security assessment module is used to ensure the safety and explainability of system operation. It achieves the explainability of model decisions through dynamic security boundary construction and hierarchical attribution analysis, and continuously improves system performance based on multi-dimensional performance evaluation and closed-loop optimization control.

[0144] According to one aspect of the present application, a lightweight distributed reinforcement learning training platform based on Ray includes a front-end display module, a back-end service module, a distributed computing module, and a message interaction middleware. The front-end display module uses a user interface built with QT technology to provide a user operation interface and real-time display of learning status and results. The back-end service module is responsible for processing requests from the front end, scheduling and managing the execution of computing tasks through a server built with the Tornado framework. The distributed computing module uses the Ray distributed framework to create and manage multiple reinforcement learning agents, which process data and model training in parallel. In terms of message interaction, redis_mq is used as a message transmission middleware between the front end, back end, and computing nodes to ensure the real-time and consistency of data. In the above structure, the overall framework is divided into four main levels. The first layer is a visual front end, which provides an interactive operation interface for users. The second layer builds a back-end service system based on the Tornado framework, which is mainly responsible for overall task management and resource management. The third layer is a notification scheduling layer. When the second layer generates a task notification, the layer selects a master node suitable for startup through load balancing technology, and uses redis_mq to send a task execution command to the fourth layer, i.e., a distributed computing cluster. Once the fourth layer receives these commands, it starts the corresponding reinforcement learning algorithm on the designated master node to perform the task.

[0145] In order to further reduce the coupling between the underlying algorithm execution cluster and other system components, and enable the algorithm cluster to be deployed and run independently of the back-end management module and the front-end display module while running on this platform, a method for generating yaml execution files is adopted, through which the algorithm cluster can independently read these execution files and obtain the required algorithm parameters to perform corresponding computing tasks. This embodiment enhances the module independence and scalability of the system, and at the same time improves the flexibility of algorithm deployment and execution.

[0146] In another embodiment of the present application, a user creates a training task and generates a task configuration file named Task.yaml. To start the training task, the task management module receives the start request and sends a notification to the notification scheduling layer. After receiving the request, the scheduling layer extracts the list of hosts registered by the user from the local database, and performs load balancing according to the task running status of the host (sorted based on the usage count) to select the most suitable host. Afterwards, the notification scheduling layer instructs the resource management module to push the Task.yaml file and related scenario files to the selected host, and starts the algorithm execution program on the host through the SSH protocol. In this embodiment, in order to ensure the state synchronization between the algorithm cluster and the back-end service and reduce their mutual coupling, a lightweight message middleware such as redis-mq is adopted. The algorithm execution process is as follows: before the algorithm starts, a message will be sent to the notification scheduling layer through redis-mq to increase the usage count of the corresponding host IP; during the execution of the algorithm task, a redis-mq message will be sent to the task management module to update the task status and persist the training evaluation data; in addition, the unstructured data files such as models and samples will be pushed to the resource management module through the SSH protocol; when the algorithm is executed, it is necessary to send a message to the notification scheduling layer through redis-mq again to reduce the usage count of the corresponding host IP. If any error that causes the task to terminate occurs during the execution process, the usage count must also be reduced through redis-mq. In order to further improve the independent operation capability, all necessary parameters are extracted through Task.yaml generated by the task management module. In this way, relying on the fixed Task.yaml template configuration file, not only can the underlying algorithm cluster be independently deployed in a lightweight manner from the backend management module and the front-end display module, but also can be adjusted flexibly in real time during the debugging process by modifying the relevant parameters in the yaml file.

[0147] This embodiment builds a lightweight, efficient, stable and easily expandable distributed learning system. The system uses QT technology to build a front-end display module, providing an interactive user interface and real-time display of learning status and results. The back-end service module is based on the Tornado framework and is responsible for processing front-end requests, executing task scheduling and management functions. The distributed computing module uses the Ray framework to support the efficient execution of distributed reinforcement learning algorithms. Message interaction in the system is implemented through redis_mq to ensure the real-time and consistency of data. At the same time, a method of using yaml configuration files is also used to support the independent operation of the underlying algorithm execution cluster and the flexibility of parameter configuration, reduce the coupling between system components, and improve the module independence and scalability of the overall architecture. This embodiment can effectively improve the execution efficiency of reinforcement learning tasks and reduce the maintenance complexity and resource consumption of the system.

[0148] The present invention realizes the high efficiency and stability of the distributed reinforcement learning training platform through multi-level feature processing, strategy optimization, parameter optimization and deployment optimization. A unified topological feature representation is generated through temporal adjacency matrix and manifold embedding mapping to ensure that the global and local structural features of the data are preserved. Based on the compressed features and quality scores, the state manifold and tangent space are constructed to generate a structured reward function, and the Wasserstein metric and policy gradient are collaboratively updated to optimize the policy set. By synchronizing global parameters, a parameter manifold hierarchical structure is constructed, the inter-layer connection is optimized, the optimized parameter structure is generated, and selective knowledge update is performed based on the knowledge graph. Based on the final compression model parameters and performance indicators, a load distribution matrix is ​​constructed, the communication cost between nodes is optimized, and a task scheduling strategy is generated to ensure efficient operation and load balancing of the system. A safe area is constructed through the Lyapunov function to generate safe training constraints, and an explanation model is generated based on feature importance and decision paths to improve the transparency and interpretability of the model. The present invention improves the performance, stability and interpretability of the distributed reinforcement learning training platform.

[0149] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A design method for a lightweight distributed reinforcement learning training platform based on Ray, characterized in that: The steps include: S1. Obtain the original data stream set, construct the time-series adjacency matrix, generate the manifold embedding map, and obtain a unified topological feature representation; represent the topological features, calculate the feature stability matrix and stability weight, and perform dynamic smoothing to obtain a stabilized feature representation; Based on the stabilized feature representation, the local probability distribution and distance-preserving metric are calculated to generate a compressed feature representation. Based on the compressed feature representation and the topological feature representation, the information retention rate and topology retention are calculated to generate a quality score and optimization suggestions. The original data stream set includes timestamps and feature vectors; S2. Based on the compressed feature representation and quality score, the state manifold and tangent space are constructed, the value function gradient and geodesic are calculated, and the structured reward function is generated; based on the pre-stored node strategy set and the structured reward function, the Wasserstein metric and strategy gradient are calculated and updated collaboratively to obtain the optimized strategy set; based on the optimized strategy set and the pre-stored historical experience pool, the trajectory entropy and importance weight are calculated, the priority queue is constructed, and the updated experience pool is obtained; Based on the updated experience pool, collect the strategy parameters and gradients of each node, calculate the Karcher mean and perform momentum correction to obtain the synchronized global parameters; S3. Based on the synchronized global parameters, the parameter manifold hierarchical structure and inter-layer mapping are constructed, and the inter-layer connections are optimized to obtain the optimized parameter structure; based on the optimized parameter structure and the pre-stored historical model set, the knowledge graph is constructed, the importance of knowledge units is calculated, and selective knowledge updates are performed to obtain the updated knowledge base; Obtain the current environment features, build the environment feature manifold and invariance measurement based on the current environment features and the updated knowledge base, and generate a generalization strategy; Based on the generalization strategy, the parameter sensitivity and importance are calculated and structured pruning and quantization optimization are performed to obtain the final compression model parameters; S4. Based on the final compression model parameters and the pre-stored performance indicators, a load distribution matrix is ​​constructed to generate an initial parameter distribution scheme; Iteratively optimize the initial parameter distribution scheme to obtain an optimized distribution scheme; based on the optimized distribution scheme, construct a load balancing objective function and generate a task scheduling strategy; Generate the final deployment configuration based on the task scheduling strategy; Step S1 is further as follows: S11, obtaining a set of original data streams from n distributed nodes, and calculating a temporal adjacency matrix based on timestamp differences and eigenvector differences of adjacent data points in the set of original data streams; Based on the time-series adjacency matrix, the characteristic spectrum and characteristic function are calculated to construct the manifold embedding map; Based on manifold embedding mapping, the original data stream set is mapped to obtain a unified topological feature representation; S12, based on the topological feature representation and the pre-stored historical feature library, calculating the cosine similarity of each feature vector to obtain a feature stability matrix; After calculating the row mean of the feature stability matrix, a softmax operation is performed to obtain the stability weight; the topological feature representation, the stability weight and the mean of the historical feature library are weighted and combined to obtain the stabilized feature representation; S13, based on the stabilized feature representation, calculate the local probability distribution of the feature vector; based on the local probability distribution, use Euclidean distance and KL divergence to construct a distance-preserving metric and calculate the distance-preserving error; initialize the compression mapping parameters, and perform gradient descent optimization on the compression mapping parameters by minimizing the distance-preserving error to obtain the optimized parameters; based on the optimized parameters, perform compression mapping on the stabilized feature representation to obtain a compressed feature representation; S14. Calculate the mutual information and feature entropy between the compressed feature representation and the topological feature representation to obtain the information retention rate; based on the compressed feature representation and the topological feature representation, calculate the distance matrix of the data points in the compressed feature space and the original feature space, and based on the distance matrix, obtain the topology preservation degree through the preset rank correlation coefficient; perform weighted combination of the information retention rate and the topology preservation degree to generate a quality score and optimization suggestions.

2. The design method of the Ray-based lightweight distributed reinforcement learning training platform according to claim 1, characterized in that: Step S2 is further as follows: S21. Based on the compressed feature representation and quality score, a state manifold is constructed in the compressed feature space, and the tangent space at each state point is calculated; based on the tangent space, the gradient of the preset value function is calculated for each state point, and the Riemann metric tensor is constructed; Based on the Riemann metric tensor, the geodesic equation is used to generate the current trajectory and obtain the geodesic distance. Based on the geodesic distance and quality score, the structured reward function is calculated and the state-action mapping relationship is generated. S22, obtain the node strategy set of the distributed nodes, calculate the Wasserstein metric between any two strategies on the strategy manifold based on the structured reward function and the node strategy set, and simultaneously calculate the policy gradient of each strategy, including the Q value and the expected term of the state value function; combine the policy gradient and the Wasserstein metric for collaborative update to obtain the optimized strategy set; S23, calculating the state-action distribution entropy of each experience trajectory based on the optimization strategy set and the pre-stored historical experience pool; Calculate importance weights based on state-action distribution entropy; Combine importance weights and structured reward functions to build a priority queue; select top-k samples from the priority queue and randomly sample samples from the historical experience pool to form an updated experience pool; S24. Based on the updated experience pool, collect the policy parameters and gradients of each distributed node; based on the policy parameters and gradients, construct an exponential mapping from the tangent space to the manifold, and calculate the Karcher mean that minimizes the sum of squares of geodesic distances between parameters; calculate the logarithmic mapping of the Karcher mean; based on the logarithmic mapping and the pre-stored historical momentum, calculate the new momentum; map the new momentum back to the parameter manifold through the exponential mapping to obtain the synchronized global parameters and consistency indicators.

3. The design method of the Ray-based lightweight distributed reinforcement learning training platform according to claim 2, characterized in that: Step S3 is further as follows: S31. Based on the synchronization global parameters and consistency index, the parameter space is divided into k abstract levels, and a reversible mapping function between adjacent levels is constructed; based on the reversible mapping function, the Frobenius norm of the mapping combination and the unit mapping is calculated to obtain the structural stability index; the Hessian matrix of the hierarchical structure is constructed, and the trace of the Hessian matrix is ​​calculated; based on the structural stability index and the trace of the Hessian matrix, the optimized parameter structure is generated; Where k is a preset natural number; S32. Based on the optimized parameter structure and the pre-stored historical model set, a knowledge graph is constructed, where the nodes of the knowledge graph represent knowledge units, and the edges represent knowledge associations, and the difference between new and old predictions is calculated; Calculate the PageRank value of each knowledge unit in the knowledge graph and combine it with its timeliness to get the importance score; calculate the knowledge retention loss based on the importance score and the difference between the new and old predictions; Based on the knowledge retention loss, selective knowledge update is performed to obtain an updated knowledge base; S33, obtaining current environment features, and mapping the current environment features to a feature manifold through a feature mapping function based on the updated knowledge base; Based on the feature manifold, the invariance measure between environmental features is calculated in the reproducing kernel Hilbert space; based on the invariance measure, the environmental variation is calculated; Generate a generalization strategy based on the regularization term of the environment variation and policy gradient; S34, based on the generalization strategy and the preset system deployment constraints, calculate the outer product of the gradient of the loss function with respect to the parameters to obtain the parameter sensitivity matrix; Based on the parameter sensitivity matrix, calculate the importance weight value of each parameter; Based on the importance weight value, structured pruning is performed to obtain the pruned parameters; Perform b-bit quantization on the pruned parameters to obtain the final compressed model parameters.

4. The design method of the Ray-based lightweight distributed reinforcement learning training platform according to claim 3 is characterized in that: Step S4 is further as follows: S41, obtaining the performance index of the system, collecting the resource state vector of each node based on the final compression model parameters and the performance index; and calculating the computational complexity of the final compression model parameters at each node based on the resource state vector; Based on the computational complexity, a load distribution matrix is ​​constructed to generate an initial parameter distribution scheme; S42, calculating the communication cost between nodes based on the initial parameter distribution scheme; Based on the communication cost, construct the node affinity matrix; Based on the node affinity matrix, the initial parameter distribution scheme is iteratively optimized to generate an optimized distribution scheme; S43. Based on the optimized distribution scheme, the model execution performance of each node is calculated; based on the model execution performance, a task scheduling matrix is ​​constructed; Based on the task scheduling matrix, construct the load balancing objective function and generate the task scheduling strategy; S44. Based on the task scheduling strategy, synchronize the distributed model and obtain the synchronization result; based on the synchronization result, calculate the synchronization overhead and gradient aggregation delay; based on the gradient aggregation delay, optimize the synchronization cycle and generate the final deployment configuration.

5. The design method of the Ray-based lightweight distributed reinforcement learning training platform according to claim 4, characterized in that: Step S11 is further as follows: S111, obtaining a set of original data streams from n distributed nodes, and extracting a timestamp sequence and a feature sequence in each data stream; Calculate the statistical moment of the feature sequence and perform standardization transformation to obtain standardized features; use a sliding window of a predetermined length to segment the standardized features into time series to generate a set of feature segments; S112, based on the feature segment set, calculating the autocorrelation matrix of each feature segment; performing eigenvalue decomposition on the autocorrelation matrix to obtain an eigenvalue sequence; based on the size of the eigenvalue sequence, selecting the first k eigenvectors to construct a projection matrix; Applying the projection matrix to the feature fragment set to perform dimensionality reduction transformation to obtain a low-dimensional representation; S113, calculating the Euclidean distance matrix between different data points based on the low-dimensional representation; Based on the Euclidean distance matrix, determine the k nearest neighbors of each data point and construct an adjacency matrix; Based on the adjacency matrix, the weight matrix is ​​calculated; Based on the weight matrix, the graph Laplacian matrix is ​​calculated; S114. Construct a timing constraint matrix based on the graph Laplacian matrix and the timestamp sequence; The graph Laplacian matrix and the timing constraint matrix are combined to construct the objective function; Based on the objective function, solve the generalized eigenvalue problem and obtain the global coordinate matrix; Based on the global coordinate matrix, construct the manifold mapping function; S115, applying the manifold mapping function to all data points to obtain an initial topological feature representation; Based on the initial topological feature representation, the mutual information matrix between different node feature representations is calculated; Based on the mutual information matrix, a consistency loss function is constructed; The consistency loss function is minimized by the gradient descent method to obtain a unified topological feature representation.

6. The design method of the Ray-based lightweight distributed reinforcement learning training platform according to claim 4, characterized in that: Step S22 is further as follows: S221, obtaining a node strategy set of a distributed node, sampling trajectories from the node strategy set, and calculating a cumulative reward for each trajectory based on a structured reward function; and calculating an advantage function for each state-action pair based on the cumulative reward; Based on the advantage function, calculate the policy gradient; S222. Based on the policy gradient, construct the geodesic on the policy distribution space and calculate the Wasserstein distance between any two policy distributions. Based on geodesics, the exponential mapping and logarithmic mapping of the strategy distribution are calculated to generate the strategy transfer path; S223, calculating the distance matrix of each pair of strategies based on the strategy transfer path and Wasserstein distance; Based on the distance matrix, construct a strategic neighbor graph; Calculate the centrality index on the strategy neighbor graph, and assign a collaborative weight to each strategy based on the centrality index; S224, based on the collaborative weight and the policy gradient, calculate the weighted average gradient; based on the weighted average gradient and the policy transfer path, update each policy parameter to obtain an optimized policy set; based on the optimized policy set, calculate the KL divergence matrix; S225. Calculate the strategy consistency index based on the optimization strategy set and the KL divergence matrix; Calculate the learning rate based on the policy consistency indicator; Based on the learning rate, the performance indicators of each strategy are calculated to generate the final optimized strategy set.

7. The design method of the Ray-based lightweight distributed reinforcement learning training platform according to claim 4, characterized in that: Step S31 is further as follows: S311, based on the synchronization global parameters and consistency indicators, the parameter space is divided into k subspaces; the parameter gradient of each subspace is calculated; based on the parameter gradient, a parameter correlation matrix is ​​constructed; Perform spectral clustering on the parameter correlation matrix to obtain an initial hierarchical structure; S312, based on the initial hierarchical structure, calculating the mapping matrix between adjacent hierarchies; Calculate the condition number of each mapping matrix; Based on the condition number, an inter-layer stability index is constructed to generate an inter-layer connection strength matrix; S313, calculating the Hessian matrix of each layer based on the inter-layer connection strength matrix; Calculate the eigendecomposition of the Hessian matrix and obtain the eigenvalues ​​and eigenvectors; based on Eigenvalues ​​and eigenvectors, calculate intra-layer complexity, and generate layer complexity vectors; S314. Based on the layer complexity vector and the inter-layer connection strength matrix, construct a hierarchical optimization objective function; Calculate the hierarchy gradient of the hierarchical optimization objective function; based on the hierarchy gradient, update the hierarchy to generate an optimized hierarchy; S315, based on the optimized hierarchical structure, calculating the information flow matrix between the hierarchies; Based on the information flow matrix, a structural stability index is constructed; Based on the structural stability index, normalization is performed on the parameters of each layer to obtain a standardized hierarchical structure as the optimization parameter structure.

8. The design method of the Ray-based lightweight distributed reinforcement learning training platform according to claim 4, characterized in that: Also includes: S51. Based on the final deployment configuration and the final compression model parameters, construct a Lyapunov function, and calculate the parameter gradient norm and update step size; Based on the parameter gradient norm and update step size, a safe region is constructed to generate safe training constraints; S52, calculating feature importance based on safety training constraints; Based on the feature importance, a decision explanation diagram is constructed to extract the key decision path and generate a decision explanation model; S53, calculate the performance indicators of the decision explanation model in each scenario, construct the evaluation indicator space, and generate multi-dimensional indicators; Based on multi-dimensional indicators, calculate the comprehensive score and generate a performance evaluation report; S54. Based on the performance evaluation report, construct feedback optimization goals and calculate parameter adjustment directions; Based on the parameter adjustment direction, an adaptive learning rate is used to generate parameter update suggestions and feed them back to step S3.

Images