Password resource allocation method and system based on attention mechanism and residual network

Abstract

The invention discloses a password resource allocation method and system based on an attention mechanism and a residual network, and relates to the field of password allocation based on a deep Q network. The method and the device aim at meeting high-concurrency and multi-task complex network decision-making requirements. The password service scene covers multiple task types, including encryption and decryption operation, large-scale signature verification, key management and cross-service security calculation. The tasks put forward extremely high real-time performance and accuracy requirements for computing resources, communication bandwidth and task scheduling of the system. According to the method, the deep reinforcement learning method is introduced, the characteristics of different password service tasks are dynamically adapted, intelligent decision making can be carried out based on the real-time system state and service requirements, and therefore the resource utilization efficiency is remarkably improved. Meanwhile, by optimizing task path planning, system time delay is reduced, load balancing is achieved, and the performance stability and task response capacity of the password service in a high-concurrency scene are effectively guaranteed.

Description

Technical Field

[0001] The present application relates to the field of password allocation, and in particular to a password resource allocation method and system based on attention mechanism and residual network. Background Art

[0002] In modern network environments, cryptographic services, as an important means of security protection, require efficient resource scheduling to meet the needs of different users. However, since cryptographic services require a variety of different computing resources, and these resources are distributed on a variety of different cryptographic computing devices, the resource scheduling problem becomes extremely complex and challenging. Traditional resource scheduling algorithms such as SJF (Shortest Job First), SRTF (Shortest Remaining Time First), FCFS (First Come First Served), etc., although they perform well in certain specific scenarios, they often seem to be unable to cope with dynamic and uncertain network environments.

[0003] Existing solutions all ignore two issues: 1) Limitations of using average computing power to predict time: In cryptographic service scenarios, directly using average computing power to predict task completion time may lead to errors. The complexity of cryptographic calculations often varies greatly due to differences in algorithms and hardware support, and the average computing power cannot accurately reflect the performance of different devices when executing specific algorithms. This ignores fine-grained performance modeling and prediction, especially the analysis of differences between different hardware platforms and algorithm types. 2) Dynamic adjustment and adaptive scheduling strategies: Research focuses on static scheduling strategy design, while ignoring the need to dynamically adjust scheduling strategies based on real-time load conditions. In complex cloud environments, resource loads and demands are constantly changing, and static strategies are difficult to cope with changing environments. For cryptographic service tasks that need to run for a long time, adaptive scheduling methods have not been fully considered, especially when faced with unpredictable task demands and burst loads.

[0004] Therefore, in the dynamic cryptographic service deployment scenario, a dynamic and instant scheduling method needs to be considered. The scheduling of cryptographic services faces three main problems: first, the complexity of multiple resources and multiple servers leads to high dimensions of state and action space; second, the dynamic changes of demand in time and space; third, the cooperation and competition of resources among multiple cryptographic services. In view of the complexity of high-dimensional state and action space, reinforcement learning, especially deep reinforcement learning (DRL), uses deep neural networks to approximate high-dimensional space and effectively copes with multi-dimensional decision-making problems that are difficult to handle with traditional optimization methods. Secondly, reinforcement learning can adapt to the dynamic changes of demand in time and space in cryptographic services in real time with its continuous "exploration-exploitation" mechanism, and dynamically adjust resource allocation strategies to ensure the optimal performance of the system under different conditions. In addition, in view of the resource cooperation and competition problems among multiple cryptographic services, multi-agent reinforcement learning promotes effective resource coordination among multiple services and optimizes overall performance by designing a reasonable reward mechanism. This makes reinforcement learning an ideal choice for solving the problem of cryptographic service scheduling. Summary of the invention

[0005] In response to the problem of cryptographic service scheduling, the present application provides a cryptographic resource allocation method and system based on an attention mechanism and a residual network. By introducing reinforcement learning methods into cryptographic service resource allocation, cryptographic resources are allocated through efficient deep reinforcement learning based on an attention mechanism and a residual network, thereby achieving multi-objective optimization of cryptographic resource decisions to cope with resource scheduling problems in complex cloud computing environments.

[0006] The first aspect of the present application provides a cryptographic resource allocation method based on an attention mechanism and a residual network, the method comprising the following steps:

[0007] Step 1: Model the cryptographic service computing resources to be allocated using a Markov decision process to obtain a cryptographic resource allocation model;

[0008] Step 2: Use the attention mechanism and residual network to build the network structure of the password resource allocation model based on the Deep Q-Network (DQN), and perform deep reinforcement learning to optimize the network parameters of the built network structure to obtain a trained password resource allocation model;

[0009] Then, based on the trained cryptographic resource allocation model, the cryptographic resource allocation results of the cryptographic service computing resources to be allocated are obtained to achieve multi-objective optimization of decision-making.

[0010] Furthermore, the cryptographic resource allocation model is represented by a five-tuple (S, A, P, R, γ);

[0011] Where S represents the state space of the model, which includes all possible state sets in the environment. In this application, the environment includes tasks T, services SRV, and service nodes Node;

[0012] A represents the action space of the model, which contains the set of all actions that the agent can take in different states;

[0013] P represents the state transition probability of the model, that is, in state s∈S, after executing action α∈A, it transfers to the next state s ′ ∈S probability P(s′|s,a);

[0014] R represents the reward function of the model, that is, the immediate reward r = R(s, a) obtained by the agent of the deep Q network after performing action α∈A in state s∈S;

[0015] γ represents the discount factor of the model, which is used to control the importance of long-term rewards and short-term rewards.

[0016] Furthermore, in step 1, the constructed cryptographic resource allocation model includes two neural networks with the same network structure: an evaluation network and a target network; wherein the evaluation network (also called the value network) is used to predict the expected return value of taking a certain action according to the current environmental state, i.e., the Q value; the target network is used to calculate the target Q value; the parameters of the evaluation network are continuously updated with deep reinforcement learning, and the parameters of the target network are regularly updated synchronously from the evaluation network;

[0017] When performing deep reinforcement learning on the cryptographic resource allocation model in step 2, it specifically includes:

[0018] The deep Q-network agent interacts with the environment and collects samples (s,a,r,s ′ ) and stored in the experience replay pool of the deep Q network;

[0019] Randomly sample a batch of data from the experience replay pool and use the evaluation network to predict the Q value; and calculate the target value based on the target Q value output by the target network: y = r + γQ target , where Q target The target Q value output by the target network;

[0020] Update the evaluation network parameters by minimizing the error between the target value y and the predicted Q value of the evaluation network, and then continue deep reinforcement learning on the next batch of sampled data based on the updated evaluation network;

[0021] When the parameter update cycle of the target network is met, the parameters of the evaluation network are copied to the target network. For example, each time the agent completes a specified number of interactions, the parameter update of the target network is triggered.

[0022] Furthermore, the network structures of the evaluation network and the target network include: input layer, attention mechanism layer, residual layer, fully connected layer and output layer;

[0023] Among them, the input layer is used to receive the current environment state and construct a high-dimensional state vector in, represents the state vector of the i-th cryptographic service node at time t, n represents the number of cryptographic service nodes; each The status information includes: task information requested by the user, equipment performance parameters of the current node, status information of the current node, such as node load and node geographic location, etc.

[0024] The attention mechanism layer is used to calculate the correlation between each password service node to quantify the importance of the password service node and obtain the importance of the password service node;

[0025] The residual layer enhances the depth of the network by stacking residual blocks and learns the relationship between node status and task allocation strategy;

[0026] The fully connected layer is used to map the features output by the residual layer into high-dimensional features;

[0027] The output layer generates the expected reward value for each possible task assignment.

[0028] Furthermore, the attention mechanism layer is specifically:

[0029] The state vector s t Mapped to the query matrix Q, key matrix K and value matrix V:

[0030] Q=W Q s t ,K=W K s t ,V=W V s t

[0031] Among them, W Q , W K and W V They are respectively the learnable query weight matrix, key weight matrix and value weight matrix;

[0032] The attention weight matrix is ​​calculated through the attention mechanism:

[0033]

[0034] The weighted state vector is obtained based on the attention weight matrix:

[0035]

[0036] Among them, softmax(·) is the normalized exponential function, d k is the key weight matrix W K The dimension of is the weighted state vector.

[0037] Furthermore, each residual block of the residual layer is composed of at least two fully connected layers, and the output features of each residual block are the nonlinear transformation of at least two fully connected layers plus the output features of the previous residual block; wherein the input of the first residual block is the weighted state vector The input of the remaining residual blocks is the output features of the previous residual block.

[0038] Furthermore, in the fully connected layer, nonlinear transformation is performed through multiple layers of fully connected layers to obtain high-dimensional features:

[0039]

[0040] Among them, W f (f=1,2,…,n′) is the weight matrix of the fth fully connected layer, b f is the corresponding bias vector, It is the high-dimensional feature representation of the output.

[0041] Furthermore, the processing of the output layer includes:

[0042] Calculate the Q value of the output layer according to the Q value function:

[0043] The agent chooses the best action: a * = argmax α Q(s t ,a);

[0044] Among them, W q , b q are the weight matrix and bias vector of the output layer respectively, is the high-dimensional feature representation of the final output of the fully connected layer. The Q value represents the expected return of assigning a task to a specific service node in the current state. * It represents the task allocation strategy that can be selected to maximize the expected return in the current state, that is, the optimal action.

[0045] The second aspect of the present application provides a cryptographic resource allocation system based on an attention mechanism and a residual network, comprising a central server and a plurality of cryptographic service resource servers (i.e., cryptographic service resource nodes);

[0046] Among them, cryptographic service resource nodes include but are not limited to server cryptographic machines, signature servers, digital certificate servers, etc.;

[0047] The user initiates a cryptographic service task application to the central server through the network, and the central server sends the status vector of each cryptographic service resource node to the central server. The target network is input into the trained cryptographic resource allocation model, and the optimal allocation strategy is obtained based on its output, and then the task request is allocated to the cryptographic service resource node corresponding to the optimal allocation strategy; wherein the state vector of the cryptographic service resource node It includes the task information requested by the user, the device performance parameters of the current node and the status information of the current node. The subscript t is used to identify the time, and the superscript i is used to identify the cryptographic service resource node. Each assigned cryptographic service resource node responds to the cryptographic service task application and returns the response result to the user who initiated the cryptographic service task application through the central server, such as providing the service request user with relevant data resource downloads and updates, so as to realize dynamic deployment and flexible allocation of resources.

[0048] The technical solution provided by this application brings at least the following beneficial effects:

[0049] Combining the self-learning ability of deep reinforcement learning, the decision-making problem of cryptographic service computing resources is modeled as a Markov decision process, which realizes the adaptive allocation of cryptographic service resources under dynamic load and multiple cryptographic algorithm requirements to optimize resource utilization efficiency. An improved attention-residual enhanced multi-agent deep Q network algorithm is designed and proposed. On the Q network of each DQN agent, an attention mechanism layer is introduced to capture the dependency between different parts of the state, accelerating the convergence speed and stability of the algorithm. At the same time, a residual network is added to avoid the gradient vanishing problem in the deep network, thereby improving the effect of training the deep network. Multi-objective optimization of cryptographic resource decision-making is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0051] Figure 1 The overall flow chart of the cryptographic resource allocation method based on the attention mechanism and residual network provided in the embodiment of the present application;

[0052] Figure 2 This is a cloud cryptographic service deployment diagram for an embodiment of the present application.

[0053] Figure 3 This is a diagram describing the cryptographic service situation of an embodiment of the present application.

[0054] Figure 4 This is a framework diagram of the cryptographic resource allocation method based on the attention mechanism and residual network in an embodiment of the present application.

[0055] Figure 5 Schematic diagram of the network structure of the attention-residual enhanced multi-agent deep Q-network algorithm of an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions of the embodiments of the present application will be described in detail and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the embodiments described with reference to the drawings are exemplary and are intended to be used to explain the present application, and cannot be understood as limiting the present application.

[0057] An embodiment of the present application provides a cryptographic resource allocation method based on an attention mechanism and a residual network, which is used to solve the cryptographic service scheduling problem. The method of the present application introduces a reinforcement learning method into the cryptographic service resource allocation to cope with the resource scheduling problem in a complex cloud computing environment.

[0058] This application combines the self-learning ability of deep reinforcement learning to model the decision-making problem of cryptographic service computing resources as a Markov decision process, realizing the adaptive allocation of cryptographic service resources under dynamic loads and multiple cryptographic algorithm requirements to optimize resource utilization efficiency. An improved attention-residual enhanced multi-agent deep Q network algorithm (i.e., cryptographic resource allocation model) is designed and proposed. On the Q network of each DQN agent, an attention mechanism layer (Attention Mechanism Layer) is introduced to capture the dependencies between different parts of the state and accelerate the convergence speed and stability of the algorithm. At the same time, a residual network (Residual Network) is added to avoid the gradient vanishing problem in the deep network, thereby improving the effect of training the deep network. Multi-objective optimization of cryptographic resource decisions is achieved.

[0059] First, the decision problem is modeled as a Markov decision process, represented by a five-tuple (S, A, P, R, γ). The modeling method is as follows:

[0060] S: State Space, which represents the set of all possible states in the environment. In this application, the environment is composed of tasks T, services SRV, and service nodes Node.

[0061] T: Task set, defined as Ts = {T 1 ,T 2 ,…,T n}, where n represents the number of tasks, each T iA set of functional operations or activities designed to achieve a specific goal. These operations may involve the collaboration of multiple services. Tasks include but are not limited to electronic signatures, user registration, user login, data exchange, and data storage.

[0062] SRV: Service set, defined as SRVs = {SRV 1 ,SRV 2 ,…,SRV m}, m is the number of services, where each service SRV j It is an independent component that provides specific functions or resources and can be reused by multiple tasks, emphasizing function realization and modularization. Services include but are not limited to CA services, certificate repositories, identity authentication services, signature services, and various security and encryption services.

[0063] Node: Service node set, defined as Nodes = {Node 1 ,Node w ,…,Node w}, where w represents the total number of service nodes. Each service node Node i (i=1,2,…,w) is a server that can execute several services. After being split into the service set SRV, the task set T reaches the service node cluster through network routing. The servers include but are not limited to server cryptographic machines, terminal cryptographic machines, digital certificate servers, signature servers, etc. Services must be executed on corresponding servers, and different types of servers can perform different services and performance. For example, compared with terminal cryptographic machines, server cryptographic machines have added encryption and decryption functions of specific algorithms; while terminal cryptographic machines have local management and log management functions that server cryptographic machines do not have. By selecting appropriate service nodes, the execution efficiency and resource utilization of services can be optimized.

[0064] A: Action Space, which represents the set of all actions that the agent can take in different states. The actions of the agent are intended to influence the environment, thereby achieving optimal allocation of resources and efficient execution of tasks. In this application, the definition of action space A is as follows:

[0065] A 1 : Service allocation action, the agent can decide to assign a single service SRV j or a set of service SRVs = {SRV a ,SRV b ,…} assigned to the corresponding service nodes Nodes = {Node 1 ,Node 2 ,…,Node w} to meet the task T i demand.

[0066] A 2 : Task sequence adjustment action, the agent can modify the combination Ts of the current task sequence to optimize the execution order of tasks. This includes adjusting the priority of tasks, executing tasks in parallel, and replanning the task process.

[0067] A 3 : Parameter optimization action, the agent can adjust the parameter settings of the service or task, such as the performance parameters of the service, the resource requirements of the task, etc., to achieve the best operating effect.

[0068] A 4 : Collaborative interaction, agents can facilitate the collaborative work between different services or tasks, establish service combinations or task chains, and enhance the functionality and flexibility of the system. For example, task T 1 (User Registration) and Task T 2 (Data storage) all need to call service SRV 1 (key management), the intelligent agent can concentrate the two services on the same service node, reduce the delay and cost of data transmission, and reduce repeated calls; at the same time, optimize the layout of tasks and services, promote parallel processing, and speed up task completion.

[0069] A 5 : Fault recovery action. When a service or node failure is detected, the agent can take actions such as switching to backup services and rerouting tasks to ensure the stability and reliability of the system.

[0070] P: Transition Probability, that is, in state s∈S, after executing action α∈A, it will transfer to the next state s ′ The probability P(s∈S ′ |s,a). In the algorithm, the state transition probability is controlled by the evaluation network.

[0071] R: Reward Function, that is, the immediate reward r = R (s, a) obtained by the agent after performing action α ∈ A in state s ∈ S.

[0072] γ: Discount Factor, used to control the importance of long-term rewards and short-term rewards, with a value range of [0,1].

[0073] Any other tasks or operations that are basically the same as the above models in terms of function, means and effect and can be associated with by ordinary technicians in this field without creative work shall be deemed as equivalent features and included in the scope of protection.

[0074] In the second aspect, the process is optimized by the attention-residual enhanced multi-agent deep Q network algorithm to achieve multi-objective optimization of decision making. In the first aspect, the reward function directly affects the current update of the Q value, so that the Q value can reflect the quality of the current action. The Q network update includes the following steps:

[0075] Step 1: The input layer receives the current system's task information including user requests, performance parameters of each cryptographic service node, current load of each node, and geographic location, etc., to form a high-dimensional state vector in, Represents the state vector of the i-th node at time t.

[0076] Step 2: The attention mechanism layer calculates the correlation between each service node and determines the importance of the node. t Perform the following steps:

[0077] Step 2-1: Transform the state vector s t Mapping to matrices Q, K, V:

[0078] Q=W q s t ,K=W K s t ,V=W V s t

[0079] Step 2-2: Calculate the attention weight matrix through the attention mechanism:

[0080]

[0081] Step 2-3: Get the weighted state vector:

[0082]

[0083] Among them, Q represents the query matrix, K represents the key matrix, V represents the value matrix, and W Q ,W K ,W V represents the learnable weight matrix, Attention(Q,K,V) is the attention weight matrix, softmax(·) is the normalized exponential function, d k W K The dimensions of the matrix, is the weighted state vector.

[0084] Step 3: The residual layer enhances the depth of the network by stacking residual blocks and learning more complex relationships between node states and task allocation strategies. Each residual block consists of two or more fully connected layers. The following steps are performed for the residual block:

[0085] Step 3-1: Calculate the residual block That is, the output of each residual block is the sum of the residual block input and the output of the last fully connected layer of the residual block;

[0086] Step 3-2: Stack multiple residual blocks and output

[0087] Among them, F(·) represents the nonlinear transformation of the residual block, F 1 ,F 2 ,…,F k is a nonlinear transformation of multiple residual blocks, k represents the number of residual blocks included from the residual layer, Represents the state vector processed by the deep network, that is, the output feature of the residual layer.

[0088] Step 4: The fully connected layer further combines the high-dimensional features extracted in steps 2 and 3 to provide refined high-level features for task allocation decisions. The linear transformation formula of the fully connected layer is

[0089]

[0090] After a nonlinear transformation is performed through multiple layers of fully connected layers, a compressed high-level feature representation is obtained.

[0091]

[0092] Among them, ReLU(·) is the activation function, W f (f=1,2,…,n) is the weight matrix of the fully connected layer f, b is the bias vector, and n is the number of fully connected layers.

[0093] Step 5: The output layer generates the expected reward value for each possible task assignment. For the high-level feature representation, the following steps are performed:

[0094] Step 5-1: Calculate the Q value of the output layer:

[0095]

[0096] Step 5-2: The agent selects the optimal action according to the following strategy:

[0097] a * = argmax α Q(s t ,a)

[0098] The Q value represents the expected return of assigning a task to a specific service node under the current state, and a * It represents the task allocation strategy that the system can choose to maximize the expected return in the current state.

[0099] Example

[0100] The embodiment of the present application provides a cryptographic resource allocation method based on an attention mechanism and a residual network, such as Figure 1 As shown in the figure, this method first models the cryptographic resources to be allocated as a Markov decision process as the input layer of the environment-forming algorithm network. The agent selects an action based on the current state to act on the environment to obtain the next state and corresponding reward feedback. The interaction data will also be stored in the experience replay pool. At the same time, data will be randomly sampled from the experience replay pool for training and evaluation of the algorithm network to enhance data utilization and reduce the correlation between samples. Finally, the trained model (the model saved when the algorithm network meets the training convergence conditions) will be used to update the decision-making behavior of the agent and interact with the environment again to form a closed loop of cyclic optimization.

[0101] Among them, Q value is a function that represents the expected reward value of taking an action in a given state. It represents the long-term cumulative reward value that an agent can obtain by taking an action in a certain state. Reward is a signal that guides learning in reinforcement learning tasks. It represents the immediate reward after the agent takes an action in a certain situation, that is, the reward obtained at the current moment. Therefore, the difference between Q value and reward is that Q value is the expected value of a long-term cumulative reward, while reward refers to the reward value obtained at the current moment.

[0102] The experience replay pool (Replay Buffer) is an experience replay mechanism in the deep Q network. Its main function is to store past experience for random sampling when training the neural network. The rewards of each state and action are interrelated, so in continuous training, adjacent samples will have a large correlation, resulting in poor update of the neural network. Using the experience replay pool can break these correlations, making the training samples more independent, thereby improving learning efficiency.

[0103] like Figure 2 As shown in the figure, in the cloud cryptographic service, cryptographic resources are stored on various computing servers. Users initiate cryptographic tasks and request computing cluster resources. The cloud cryptographic service consists of geographically distributed users, a central server, and cryptographic service resource servers (i.e., cryptographic service resource nodes, which include but are not limited to server cryptographic machines, signature servers, digital certificate servers, etc.). Both users and cryptographic service resource nodes are dynamically distributed. Users initiate cryptographic service task applications to the central server through the network. The central server reasonably and dynamically allocates task requests to various service computing nodes based on the current performance and load status of each service node. The service node provides relevant data resource downloads and updates to the service requesting user, realizing dynamic deployment and flexible allocation of resources.

[0104] When a user submits encryption or decryption related tasks to the cloud cryptographic service system, they first communicate with the central server to explain the required service type, resource requirements, data size, expected completion time and other information. The central server monitors the utilization and performance of each resource of the distributed nodes in real time, and makes scheduling and allocation decisions based on the algorithm given in this patent, and sends scheduling instructions and related data to the nodes. After obtaining the data and instructions, the node executes the cryptographic operation required by the user, reports the results and status to the central server, and then the central server returns them to the user.

[0105] like Figure 3 As shown, at time point t, the set of available service nodes in the system is represented by Nodes = {Node 1 ,Node 2 ,…,Node m}, where the number of nodes is m, and the task request set is T = {T 1 ,T 2 ,…,T n}, task number n. Each task request needs to be split into multiple service SRVs i s={SRV i1 ,SRV i2 ,…,SRV iw}(i=1,2,…,n), these services need to reach different server nodes in turn j (j=1,2,…,m) for processing. It is worth noting that all task requests are independent of each other and have no priority relationship.

[0106] In one embodiment, see Figure 4 As shown in the figure, the system consists of an environment, an agent, and a neural network. The environment represents an external system in which multiple agents interact, which is responsible for receiving the action input of the agent and feeding back the corresponding state and reward information. A group of agents independently learn and optimize their behavior strategies by interacting with the environment. Each agent predicts the Q value of each possible action through the policy network (evaluation network) based on the state and reward information obtained from the environment, thereby deciding the next action.

[0107] The network structure includes two neural networks: the evaluation network and the target network. The evaluation network, as the core of the algorithm, is used to predict the Q value of the action based on the current state of the environment, and continuously updates its parameters during the training process to approximate the optimal Q value function. In contrast, the target network is used to calculate the target Q value, and its parameters are updated as the parameters of the evaluation network at a certain frequency (the parameter update is slower than the evaluation network), thereby improving the stability of training and avoiding training oscillations or divergence problems caused by frequent updates of the evaluation network. The value network parameters are updated frequently, and the target value changes too much, making it difficult for the optimization process to converge. The target network provides a stable reference target, allowing the value network to learn in a more stable environment.

[0108] Temporal difference is a method used to estimate the value function of a policy. Its core idea is to use the value estimate of future action choices to update the value estimate of the current action choice. The similarity between the temporal difference method and Monte Carlo is that it can learn from sample data without knowing the environment in advance. When using the policy to interact with the environment, we can use the temporal difference algorithm to update the state value estimate for each sampling step. The temporal difference algorithm uses the estimated value of the action value at the next moment, and it will eventually converge to the value function of the target policy.

[0109] The target network and the value network structure are consistent, which ensures the same expression ability of the calculated target value and the predicted value, simplifies parameter copying and design, and ensures the stability and convergence of training. In implementation, this consistency usually means that the two use the same deep neural network architecture (such as the number of layers, activation functions, convolutional layers, etc.). The only difference is that the parameters of the value network will be continuously updated during training; while the parameters of the target network will only be periodically updated synchronously from the value network.

[0110] First, the decision problem is modeled as a Markov decision process, represented by a five-tuple (S, A, P, R, γ). The modeling method is as follows:

[0111] S: State Space, which represents the set of all possible states in the environment. In this application, the environment includes tasks T, services SRV and service nodes Node.

[0112] Among them, the task T and service SRV are described as follows:

[0113] T: Task set, defined as Ts = {T 1 ,T 2 ,…,T n}, where each T iA set of functional operations or activities designed to achieve a specific goal. These operations may involve the collaboration of multiple services. Tasks include but are not limited to electronic signatures, user registration, user login, data exchange, and data storage.

[0114] SRV: Service set, defined as SRVs = {SRV 1 ,SRV 2 ,…,SRV m}, where each service SRV j It is an independent component that provides specific functions or resources and can be reused by multiple tasks, emphasizing function realization and modularization. Services include but are not limited to CA (Certificate Authority) services, certificate warehouses, identity authentication services, signature services, and various security and encryption services.

[0115] The mapping relationship between tasks and services is given by a predefined mapping table. In this embodiment, the mapping relationship shown in Table 1 is adopted.

[0116] Table 1 Mapping relationship between tasks and services

[0117]

[0118] Node: Service node set, defined as Nodes = {Node 1 ,Node 2 ,…,Node w}, where w represents the total number of service nodes. Each service node Node i (i=1,2,…,w) is a server that can execute several services. After being split into the service set SRV, the task set T reaches the service node cluster through network routing. The servers include but are not limited to server cryptographic machines, terminal cryptographic machines, digital certificate servers, signature servers, etc. Services must be executed on corresponding servers, and different types of servers can perform different services and performance. For example, compared with terminal cryptographic machines, server cryptographic machines have added encryption and decryption functions of specific algorithms; while terminal cryptographic machines have local management and log management functions that server cryptographic machines do not have. By selecting appropriate service nodes, the execution efficiency and resource utilization of services can be optimized.

[0119] The state space is represented as a two-dimensional vector (k,n s ), where k is the number of nodes in the network, n s ={menorys s ,load s ,SRVs s ,dist s} represents a vector of four state parameters for each node:

[0120] 1)Storage and forwarding capability (memory s ):

[0121] Refers to the resource capacity of a node for storing and forwarding data. This parameter can be quantified as a fixed storage space (e.g., in bytes) or a dynamically allocated capacity, reflecting the cache capacity available to the node in data processing.

[0122] 2) Load capacity s ):

[0123] Indicates the current load status of the node, including the ability to process tasks and the proportion of load pressure. This parameter is represented by the node's computing resource occupancy rate, task processing rate or queue length indicator, and the specific value can be dynamically updated over time.

[0124] 3) Service Serial Numbers (SRVs) s ):

[0125] Describes the service sequence set currently processed by the node, in the form of the service set associated with the node, identifying the current service status of the node in the network.

[0126] 4) Distance between nodes (dist s ):

[0127] The distance vector between node s and other nodes, dist s ={d s,1 ,d s,2 ,…,d s,k},d s,s =0.

[0128] A: Action Space, which represents the set of all actions that the agent can take in different states. The actions of the agent are intended to influence the environment, thereby achieving optimal allocation of resources and efficient execution of tasks. In this application, the definition of action space A is as follows:

[0129] A 1 : Service allocation action, the agent can decide to assign a single service SRV j or a set of service SRVs = {SRV a ,SRV b ,…} assigned to the corresponding service nodes Nodes = {Node 1 ,Node 2 ,…,Node w} to meet the task T i demand.

[0130] A 2: Task sequence adjustment action, the agent can modify the combination Ts of the current task sequence to optimize the execution order of tasks. This includes adjusting the priority of tasks, executing tasks in parallel, and replanning the task process.

[0131] A 3 : Parameter optimization action, the agent can adjust the parameter settings of the service or task, such as the performance parameters of the service, the resource requirements of the task, etc., to achieve the best operating effect.

[0132] A 4 : Collaborative interaction, agents can facilitate the collaborative work between different services or tasks, establish service combinations or task chains, and enhance the functionality and flexibility of the system. For example, task T 1 (User Registration) and Task T 2 (Data storage) all need to call service SRV 1 (key management), the intelligent agent can concentrate the two services on the same service node, reduce the delay and cost of data transmission, and reduce repeated calls; at the same time, optimize the layout of tasks and services, promote parallel processing, and speed up task completion.

[0133] A 5 : Fault recovery action. When a service or node failure is detected, the agent can take actions such as switching to backup services and rerouting tasks to ensure the stability and reliability of the system.

[0134] In this embodiment, the dimension of the action space is the number of edges in the network, e a ,When selecting an action, it is necessary to calculate the ,connection edges under the service node where the current service is located as the ,action selection set.

[0135] P: Transition Probability, that is, in state s∈S, after executing action α∈A, it will transfer to the next state s ′ The probability P(s∈S ′ |s,a). In the algorithm (i.e., the cryptographic resource allocation model), the state transition probability is controlled by the evaluation network.

[0136] R: Reward Function, that is, the immediate reward r = R (s, a) obtained by the agent after executing action α ∈ A in state s ∈ S. In the algorithm, the reward function includes the path length reward len reward 、Load reward load reward , load balancing reward balance reward , if the three rewards are added together to get the total reward, the specific rewards are:

[0137] len reward:This reward is evaluated based on the path length of data transmission in the network. The path length reward is determined by the node's storage and forwarding capacity memory s and the distance between nodes dist s Nodes with higher storage and forwarding capabilities can cache more data, reducing the number of data transmissions and delays in the network. reward Reward those who choose high memory s The action of connecting nodes and optimizing path length is aimed at improving the efficiency and speed of data transmission.

[0138] , where load s is the load capacity of the target node after executing action a, λ 1 is the weight coefficient of the load, and α is an exponential parameter that adjusts the degree of load influence. This reward item aims to minimize the load of the node and encourage the assignment of tasks to nodes with lower current load to avoid overload. s Reflects the resource usage of the node, including CPU usage, memory usage, etc.

[0139]

[0140] Among them, λ 2 is the weight coefficient of load balancing, σ load is the load capacity of all nodes in the current network s The standard deviation of w is the total number of service nodes, load i is the load capacity of the ith node, The average value of all node load capacities. This reward is designed to minimize the difference in load between all nodes and encourage load balancing. By reducing the standard deviation of node load capacity, the system can avoid overloading some nodes while other nodes are idle, thus improving overall performance. This item is related to the node's service sequence SRV s This is relevant because the allocation of service sequences affects the load on the nodes.

[0141] γ: Discount Factor, used to control the importance of long-term rewards and short-term rewards, with a value range of [0,1].

[0142] Second, as Figure 5 As shown, the process is optimized by the attention-residual enhanced multi-agent deep Q network algorithm to achieve multi-objective optimization of decision making. In the first aspect, the reward function directly affects the current update of the Q value, so that the Q value can reflect the quality of the current action. The update function of the Q network is:

[0143]

[0144] Among them, α is the learning rate, which controls the step size of the update. γ is the discount factor, which measures the importance of future rewards. γmax a′ Q(s ′ ,q ′ ) is the maximum Q value among all possible actions a′ in the new state s′, and r is the reward function r=R(s,a) described in the first aspect.

[0145] Q network update includes the following steps:

[0146] Step 1: The input layer receives the current system's task information including user requests, performance parameters of each cryptographic service node, current load of each node, and geographic location, etc., to form a high-dimensional state vector in, Represents the state vector of the i-th node at time t.

[0147] Step 2: The attention mechanism layer calculates the correlation between each service node and determines the importance of the node. t Perform the following steps:

[0148] Step 2-1: Transform the state vector s t Mapping to matrices Q, K, V:

[0149] Q=W q s t ,K=W k s t ,V=W v s t

[0150] Step 2-2: Calculate the attention weight matrix through the attention mechanism:

[0151]

[0152] Step 2-3: Get the weighted state vector:

[0153]

[0154] Among them, Q represents the query matrix, K represents the key matrix, V represents the value matrix, and W q ,W k ,W v represents the learnable weight matrix, Attention(Q,K,V) is the attention weight matrix, softmax(·) is the normalized exponential function, d k W k The dimensions of the matrix, is the weighted state vector.

[0155] Step 3: The residual layer enhances the depth of the network by stacking residual blocks and learning more complex relationships between node states and task allocation strategies. Each residual block consists of two or more fully connected layers. The following steps are performed for the residual block:

[0156] Step 3-1: Calculate the residual block

[0157] Step 3-2: Stack multiple residual blocks and output

[0158] Among them, F(·) represents the nonlinear transformation of the residual block, F 1 ,F 2 ,…,F k is a nonlinear transformation of multiple residual blocks, Represents the state vector processed by the deep network.

[0159] Step 4: The fully connected layer further combines the high-dimensional features extracted in steps 2 and 3 to provide refined high-level features for task allocation decisions. The linear transformation formula of the fully connected layer is

[0160]

[0161] Among them, W f (f=1,2,…,n) is the weight matrix of the fully connected layer, and b is the bias vector.

[0162] After a nonlinear transformation is performed through multiple layers of fully connected layers, a compressed high-level feature representation is obtained.

[0163]

[0164] Among them, the ReLU(·) function is an activation function, which is defined as ReLU(x)=max(0,x).

[0165] Step 5: The output layer generates the expected reward value for each possible task assignment. For the high-level feature representation, the following steps are performed:

[0166] Step 5-1: Calculate the Q value of the output layer:

[0167]

[0168] Step 5-2: The agent selects the optimal action according to the following strategy:

[0169] a * = argmax a Q(s t ,a)

[0170] The Q value represents the expected return of assigning a task to a specific service node under the current state, and a * It represents the task allocation strategy that the system can choose to maximize the expected return in the current state.

[0171] In summary, the cryptographic resource allocation method and system based on the attention mechanism and residual network provided in the embodiment of the present application is a resource scheduling solution suitable for cryptographic service scenarios, which is designed to meet the complex network decision-making requirements of high concurrency and multi-task. Cryptographic service scenarios cover a variety of task types, including encryption and decryption operations, large-scale signature verification, key management, and cross-service secure computing. These tasks place extremely high real-time and accuracy requirements on the system's computing resources, communication bandwidth, and task scheduling.

[0172] This application introduces deep reinforcement learning methods to dynamically adapt to the characteristics of different cryptographic service tasks, and can make intelligent decisions based on real-time system status and service requirements, thereby significantly improving resource utilization efficiency. At the same time, by optimizing task path planning, reducing system latency, and achieving load balancing, it effectively ensures the performance stability and task responsiveness of cryptographic services in high-concurrency scenarios.

[0173] In terms of technical implementation, this application combines the attention mechanism and residual network. The former accurately captures the dependencies and priorities between cryptographic service tasks, ensuring that high-priority tasks are given priority in resource competition; the latter enhances the expressive power of deep models, alleviates the gradient vanishing problem, and accelerates the convergence process of the model, thereby adapting to complex and changeable cryptographic service scenarios.

[0174] In addition, this application shows excellent performance under multi-task collaboration, which not only shortens the response time in high-density encryption and decryption and signature verification scenarios, but also optimizes the network path in key distribution and verification services, reducing the overall communication overhead of the system and the imbalance of encryption and decryption load. Through the autonomous learning ability of deep reinforcement learning, the algorithm can continuously optimize its own scheduling strategy to adapt to different scales and dynamically changing cryptographic service needs.

[0175] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0176] Any process or method description described in a flowchart or otherwise in this specification may be understood to represent a module, fragment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0177] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0178] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A cryptographic resource allocation method based on attention mechanism and residual network, characterized in that: The following steps are involved: Step 1: Model the cryptographic service computing resources to be allocated using a Markov decision process to obtain a cryptographic resource allocation model; Step 2: Use the attention mechanism and residual network to build the network structure of the password resource allocation model based on the deep Q network, and perform deep reinforcement learning to obtain a trained password resource allocation model; Then, based on the trained cryptographic resource allocation model, the cryptographic resource allocation results of the cryptographic service computing resources to be allocated are obtained.

2. The method according to claim 1, characterized in that The cryptographic resource allocation model is represented by a five-tuple (S, A, P, R, γ); Among them, S represents the state space of the model, which contains all possible state sets in the environment, including tasks T, services SRV and service nodes Node; A represents the action space of the model, which contains the set of all actions that the agent can take in different states; P represents the state transition probability of the model, that is, in state s∈S, after executing action α∈A, it transfers to the next state s ′ The probability P(s∈S ′ |s,a); R represents the reward function of the model, that is, the immediate reward r = R(s, a) obtained by the agent of the deep Q network after performing action α∈A in state s∈S; γ represents the discount factor of the model.

3. The method according to claim 2, characterized in that In step 1, the constructed cryptographic resource allocation model includes two neural networks with the same network structure: an evaluation network and a target network. The evaluation network is used to predict the expected return value of taking a certain action, i.e., the Q value, according to the current environmental state. The target network is used to calculate the target Q value. When performing deep reinforcement learning on the cryptographic resource allocation model in step 2, it specifically includes: The deep Q-network agent interacts with the environment and collects samples (s,a,r,s ′ ) and stored in the experience replay pool of the deep Q network; Randomly sample a batch of data from the experience replay pool and use the evaluation network to predict the Q value; and calculate the target value based on the target Q value output by the target network: y = r + γQ target , where Q target The target Q value output by the target network; The evaluation network parameters are updated by minimizing the error between the target value y and the predicted Q value of the evaluation network, and then the evaluation network parameters are iteratively updated based on the processing of the sampled data by the updated evaluation network; When the parameter update cycle of the target network is met, the parameters of the evaluation network are copied to the target network.

4. The method according to claim 1, characterized in that The network structures of the evaluation network and the target network include: input layer, attention mechanism layer, residual layer, fully connected layer and output layer; Among them, the input layer is used to receive the current environment state and construct a high-dimensional state vector in, represents the state vector of the i-th cryptographic service node at time t, n represents the number of cryptographic service nodes; each The status information includes the task information requested by the user, the device performance parameters of the current node and the status information of the current node; The attention mechanism layer is used to calculate the correlation between each password service node to quantify the importance of the password service node and obtain the importance of the password service node; The residual layer enhances the depth of the network by stacking residual blocks and learns the relationship between node status and task allocation strategy; The fully connected layer is used to map the features output by the residual layer into high-dimensional features; The output layer generates the expected reward value for each possible task assignment.

5. The method according to claim 4, characterized in that The current node status information includes node load and node geographic location.

6. The method according to claim 4, characterized in that The processing of the attention mechanism layer includes: The state vector s t Mapped to the query matrix Q, key matrix K and value matrix V: Q=W Q s t ,K=W K s t ,V=W V s t Among them, W Q , W K and W V They are respectively the learnable query weight matrix, key weight matrix and value weight matrix; The attention weight matrix is ​​calculated through the attention mechanism: The weighted state vector is obtained based on the attention weight matrix: Among them, softmax(·) is the normalized exponential function, d k is the key weight matrix W K The dimension of is the weighted state vector.

7. The method according to claim 4, characterized in that Each residual block of the residual layer consists of at least two fully connected layers. The output features of each residual block are the nonlinear transformation of at least two fully connected layers plus the output features of the previous residual block. The input of the first residual block is the weighted state vector The input of the remaining residual blocks is the output features of the previous residual block.

8. The method according to claim 4, characterized in that In the fully connected layer, nonlinear transformation is performed through multiple layers of fully connected layers to obtain high-dimensional features: Among them, W f (f=1,2,…,n′) is the weight matrix of the fth fully connected layer, b f is the corresponding bias vector, is the high-dimensional feature representation of the output, and ReLU() represents the activation function.

9. The method according to claim 4, characterized in that The processing of the output layer includes: Calculate the Q value of the output layer according to the Q value function: The agent chooses the best action: a * = argmax α Q(s t ,a); Among them, W q , b q are the weight matrix and bias vector of the output layer respectively, is the high-dimensional feature representation of the final output of the fully connected layer, s t Indicates the current state, a indicates the corresponding execution action, a * It represents the task allocation strategy that can be selected to maximize the expected return in the current state.

10. A cryptographic resource allocation system based on attention mechanism and residual network, characterized in that: It includes a central server and several cryptographic service resource nodes; The user initiates a cryptographic service task application to the central server through the network, and the central server sends the status vector of each cryptographic service resource node to the central server. The target network is input into the trained cryptographic resource allocation model according to any one of claims 1 to 9, and the optimal allocation strategy is obtained based on its output, wherein the state vector of the cryptographic service resource node It includes the task information requested by the user, the device performance parameters of the current node and the status information of the current node. The subscript t is used to identify the time, and the superscript i is used to identify the cryptographic service resource node. The task request is then assigned to the cryptographic service resource node corresponding to the optimal allocation strategy; each assigned cryptographic service resource node responds to the cryptographic service task application, and returns the response result to the user who initiated the cryptographic service task application through the central server, such as providing relevant data resources for download and update to the service requesting user.

Images