Task assignment privacy protection method, device, equipment and storage medium

Abstract

The present application relates to the field of mobile crowd intelligence perception technology. The present application discloses a task allocation privacy protection method, apparatus, device and storage medium, which can improve the accuracy of task allocation. The task allocation privacy protection method includes receiving a first state graph of the current environment, wherein the first state graph is constructed by at least one user attribute information and at least one task information, and each user attribute information is obtained by noise processing using the Laplace mechanism to meet differential privacy requirements; the first state graph is processed using the Noisy D3QN‑G algorithm based on the graph attention mechanism to obtain a task allocation result.

Description

Technical Field

[0001] The present application relates to the field of mobile crowd-sensing technology. More specifically, the present application relates to a task allocation privacy protection method, apparatus, device, and storage medium. Background Art

[0002] Traditional privacy-preserving task allocation methods use conventional deep reinforcement learning (DRL) methods to optimize task allocation performance metrics. Conventional deep reinforcement learning typically uses hand-crafted fixed-dimensional feature vectors to represent tasks and users (i.e., workers), and is unable to explicitly model complex relationships between entities such as users and tasks. However, the relationship between tasks and users is not simply a linear or simple feature mapping, but rather includes a variety of nonlinear and spatiotemporal dependencies, which place higher demands on the optimization of online task allocation strategies. Due to the lack of sufficient consideration of these complex dependencies, traditional deep reinforcement learning exhibits certain limitations when dealing with complex dependencies between tasks. In particular, when faced with large-scale and high-dimensional tasks, it struggles to effectively capture global dependencies, resulting in weak generalization and scalability, and in turn, inaccurate task allocation. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a task assignment privacy protection method, apparatus, device, and storage medium, which can improve the accuracy of task assignment. The embodiments of the present application are mainly achieved through the following technical solutions:

[0004] A first aspect of an embodiment of the present application provides a task assignment privacy protection method, comprising:

[0005] Receive a first state graph of the current environment, wherein the first state graph is constructed by at least one user attribute information and at least one task information, and each user attribute information is differentially private information obtained by performing noise processing using a Laplace mechanism;

[0006] The Noisy D3QN-G algorithm based on the graph attention mechanism is used to process the first state graph to obtain the task allocation result.

[0007] According to one embodiment of the present application, the task assignment privacy protection method further includes a training step of the Noisy D3QN-G algorithm based on the graph attention mechanism, and the training step of the Noisy D3QN-G algorithm based on the graph attention mechanism includes:

[0008] receiving a second state diagram of the historical environment;

[0009] Input the second state graph into the action network with the noise network in the original Noisy D3QN-G algorithm to calculate the Q value and obtain the first Q value;

[0010] Selecting the first action with the largest first Q value;

[0011] After sending the first action to the historical environment, receiving the next state diagram and reward fed back by the historical environment according to the first action;

[0012] The second state diagram, the first action, the reward, and the next state diagram are stored as an experience in an experience replay area, and all experiences in the experience replay area are prioritized;

[0013] When the number of all experiences in the experience replay area reaches or exceeds the requirement of a preset batch of training data, the experience replay area is randomly sampled with priority to obtain experience samples, and the experience samples are input into the action network to perform Q value calculation to obtain a second Q value, and the experience samples are input into the target network in the original Noisy D3QN-G algorithm to perform Q value calculation to obtain a third Q value;

[0014] Calculating a loss function based on the second Q value and the third Q value;

[0015] The first network parameters of the action network are adjusted based on the loss function, and the second network parameters of the target network are updated using a soft update strategy to obtain the Noisy D3QN-G algorithm based on the graph attention mechanism.

[0016] According to one embodiment of the present application, the step of inputting the second state diagram into the action network with the noisy network in the original Noisy D3QN-G algorithm to calculate the Q value to obtain the first Q value includes:

[0017] Setting a first weight, a second weight, a first bias, a second bias, and a noise factor;

[0018] generating a first random noise and a second random noise corresponding to the target linear layer from a standard normal distribution using a noise factor, wherein the target linear layer is any linear layer in the noise network;

[0019] Obtaining a third weight and a third bias of the target linear layer by calculation based on the first weight, the second weight, the first bias, the second bias, the first random noise, and the second random noise;

[0020] Performing forward propagation using the third weight and the third bias to obtain a noisy output corresponding to the target linear layer;

[0021] After obtaining the noisy outputs corresponding to all linear layers, Q value calculation is performed on the second state diagram to obtain the first Q value.

[0022] According to one embodiment of the present application, after obtaining the noisy outputs corresponding to all linear layers, performing Q value calculation on the second state graph to obtain the first Q value, the step of inputting the second state graph into the action network with the noisy network in the original Noisy D3QN-G algorithm to perform Q value calculation to obtain the first Q value further includes:

[0023] Updating a Q-value function of the noise network using the third weight and the third bias;

[0024] The third network parameter and the fourth network parameter of the noise network are updated by back propagation.

[0025] According to one embodiment of the present application, the step of calculating the Q value of the second state diagram to obtain the first Q value includes:

[0026] Embedding the original features of all nodes of the second state graph into user vectors and task vectors;

[0027] Using a first graph attention mechanism to process the user vector and a set of edges between user nodes and task nodes in the second state graph to obtain a user feature representation; using a second graph attention mechanism to process the task vector and a set of edges between user nodes and task nodes in the second state graph to obtain a task feature representation;

[0028] Calculate the attention factor using a preset algorithm;

[0029] Calculate a state and a second action based on the attention factor, the user feature representation, and the task feature representation;

[0030] The first Q value is calculated based on the state and the second action.

[0031] According to one embodiment of the present application, after the step of calculating and obtaining the first Q value based on the state and the second action, the step of calculating the Q value on the second state diagram to obtain the first Q value further includes:

[0032] Use the value function of the adversarial deep Q network in the original Noisy D3QN-G algorithm to estimate the overall value of the state;

[0033] The action advantage function of the adversarial deep Q network in the original Noisy D3QN-G algorithm is used to estimate the action advantage of taking the second action in the state;

[0034] The overall value and the action advantage are combined to generate a final Q-value estimate.

[0035] A second aspect of an embodiment of the present application provides a task assignment privacy protection device, comprising:

[0036] a receiving module, configured to receive a first state graph of a current environment, wherein the first state graph is constructed by at least one user attribute information and at least one task information, and each user attribute information is differentially private information obtained by performing noise processing using a Laplace mechanism;

[0037] A processing module is used to process the first state graph using the Noisy D3QN-G algorithm based on the graph attention mechanism to obtain a task allocation result.

[0038] In a third aspect of an embodiment of the present application, a terminal device is provided, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the steps of the task allocation privacy protection method provided in the first aspect of the embodiment of the present application.

[0039] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute the steps of the task allocation privacy protection method provided in the first aspect of the embodiment of the present application.

[0040] The beneficial effects of the embodiments of the present application include:

[0041] The embodiment of the present application effectively models the relationship between tasks and users through a graph structure. At the same time, the Noisy D3QN-G algorithm is introduced to fully capture the characteristics, potential interactions and global dependencies between tasks and users, thereby providing a more accurate decision-making basis for online task allocation. Specifically, the embodiment of the present application receives a first state graph of the current environment, wherein the first state graph is constructed by at least one user attribute information and at least one task information, and each user attribute information is obtained by noise processing using the Laplace mechanism to meet differential privacy requirements; the Noisy D3QN-G algorithm based on the graph attention mechanism is used to process the first state graph to obtain the task allocation result. Compared with the prior art, the embodiment of the present application can fully consider the various nonlinear, spatiotemporal dependencies and other relationships between tasks and users, thereby improving the accuracy of task allocation.

[0042] In addition, the embodiment of the present application uses the Laplace mechanism to perform noise processing on user attribute information and disturbs the user's location information, effectively ensuring the security of the user's privacy information, improving the enthusiasm of user participation to a certain extent, and promoting the long-term development of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A flowchart of a mobile crowd-sensing system in some embodiments;

[0045] Figure 2 A flowchart of the task allocation privacy protection method of the present application in some embodiments;

[0046] Figure 3 A flowchart of the task allocation privacy protection method of the present application in some embodiments;

[0047] Figure 4 A flowchart of the Noisy D3QN-G algorithm for solving online task allocation methods in this application;

[0048] Figure 5 This is a flowchart of the MGDRL framework of the present application in some embodiments;

[0049] Figure 6 A block diagram of the principle of a privacy protection device in some embodiments for task allocation of this application;

[0050] Figure 7 This is a principle block diagram of the terminal device of the present application in some embodiments. DETAILED DESCRIPTION

[0051] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0052] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0053] The terms "exemplary" or "for example" are used to indicate an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0054] The terms "comprises," "comprising," or any other variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0055] The term "deep reinforcement learning" refers to a reinforcement learning method based on deep neural networks. Deep learning techniques are used to model and optimize state and action spaces, enabling intelligent agents to make autonomous decisions and take actions in their environments. It primarily consists of three components: state representation, action selection, and reward evaluation. In deep reinforcement learning, agents optimize their decisions by learning strategies that maximize cumulative rewards. Through neural network optimization and training, agents continuously learn and improve their strategies, gradually achieving understanding and adaptation to their environments.

[0056] The term "Deep Q-Network" (DQN) refers to a reinforcement learning method based on deep learning. It uses deep neural networks to approximate the action-value function and employs experience replay and a fixed Q target to improve learning efficiency and stability. Experience replay reduces sample correlation and improves sample utilization by reusing past experience, thereby enhancing learning stability.

[0057] The term "graph attention mechanism" (GAT) refers to a mechanism used in graph neural networks (GNNs) that handles information transfer between nodes by assigning a different attention weight to each node in the graph. Unlike traditional graph convolutional networks (GCNs), the graph attention mechanism does not rely on structural features of the graph (such as the adjacency matrix). Instead, it dynamically adjusts information transfer by learning the relative importance of each node. In traditional graph convolutional networks, the information of all neighboring nodes is aggregated as equal weight; in the graph attention mechanism, the influence of each neighbor is weighted through the attention mechanism. This means that information transfer between nodes is weighted, and the weights are learned by the network and can be dynamically adjusted based on the context. The graph attention mechanism assigns different weights to each pair of edges between nodes. In this way, the network can adaptively determine which neighbors are more important for information transfer to the current node.

[0058] The D3QN algorithm, a state-of-the-art algorithm that integrates multiple reinforcement learning techniques, combines dual deep Q-networks, adversarial deep Q-networks, and prioritized experience replay. It aims to address challenges inherent in traditional deep reinforcement learning methods, such as overestimation of Q-values, inefficient exploration, unstable training, and getting stuck in local optima, thereby improving performance and accelerating convergence. The dual deep Q-network mitigates the impact of Q-value overestimation by using a target network to calculate Q-values ​​when selecting actions, rather than the current network. This enables more accurate action evaluation, avoiding the overestimation of Q-values ​​often seen in traditional DQNs, thereby improving learning stability and convergence. The adversarial deep Q-network improves its ability to model high-dimensional and complex state spaces by separately estimating state value and action advantage. Specifically, it calculates the value function (V) and action advantage function (A) for each state and then combines them to calculate the Q-value for each action. This results in more accurate state value estimates, improving decision-making efficiency and overall system performance. Prioritized experience replay prioritizes experiences based on TD error, allowing the network to focus more on learning experiences that are most helpful for model training, improving learning efficiency and avoiding repeated training of unimportant experiences.

[0059] The term "noisy network" refers to a network with adaptive noise, which introduces noise into the network weights. By dynamically adjusting the weight noise, the exploration process is made more efficient, avoiding the algorithm from falling into local optimal solutions and accelerating the training process.

[0060] Differential privacy refers to a privacy-preserving technique that adds noise to private data, preventing attackers from accurately inferring personal information after the data is released. Differential privacy techniques balance privacy protection and data availability by setting an appropriate privacy budget, thereby effectively protecting and utilizing data. Common privacy-preserving mechanisms used in differential privacy include the Laplace mechanism, exponential mechanism, random response mechanism, and Gaussian mechanism. The Laplace mechanism is a simple and widely used privacy-preserving mechanism for numerical queries. For numerical query results, the Laplace mechanism achieves differential privacy by returning a result that adds noise that satisfies a Laplace distribution.

[0061] The term "mobile crowd intelligence perception system" usually consists of three parties: users (workers), perception platforms, and task requesters. Figure 1 As shown in the figure, the perception platform, as a crowd-sensing service provider, is responsible for receiving perception task requests from task requesters, assigning appropriate tasks to workers, collecting and storing the perception data submitted by workers, and performing processes such as truth discovery. Task requesters are individuals or organizations that need perception data, who define perception tasks and request services from the perception platform. Users are the executors of tasks, who use smart terminal devices to collect data and receive corresponding rewards based on task requirements. The complete process of the mobile crowd-sensing system is as follows:

[0062] (1) The task requester requests services from the perception platform (e.g., temperature and humidity monitoring in a specified area, noise pollution monitoring on a specific road section, water quality monitoring in a specific area, and real-time data tracking of traffic accident hotspots, etc.). Figure 1 As shown in step ①.

[0063] (2) Use differential privacy technology based on the Laplace mechanism to perturb the location information of the perceived user to achieve privacy protection, refer to Figure 1 As shown in step ②.

[0064] (3) The perception platform aims to maximize the platform benefits and uses the Noisy D3QN-G algorithm for online task allocation. Figure 1 As shown in step ③.

[0065] (4) The perception worker performs the assigned perception task and uploads the perception data to the perception platform. Figure 1 As shown in step ④.

[0066] (5) The perception platform issues rewards to perception workers for completing tasks, refer to Figure 1 As shown in step ⑤.

[0067] (6) The perception platform processes the perception data and sends the final perception results to the task requester. Figure 1 As shown in step ⑥.

[0068] Based on the mobile crowd intelligence perception system, the key research part of this application is Figure 1 For the specific flow chart of steps ② and ③, please refer to Figure 2 shown.

[0069] Unless otherwise defined, all technical and scientific terms used in the specification of this application have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in the specification of this application includes any and all combinations of one or more of the relevant listed items.

[0070] Traditional privacy-preserving task allocation methods use conventional deep reinforcement learning (DRL) methods to optimize task allocation performance metrics. Conventional deep reinforcement learning typically uses hand-crafted fixed-dimensional feature vectors to represent tasks and users (i.e., workers), and is unable to explicitly model complex relationships between entities such as users and tasks. However, the relationship between tasks and users is not simply a linear or simple feature mapping, but rather includes a variety of nonlinear and spatiotemporal dependencies, which place higher demands on the optimization of online task allocation strategies. Due to the lack of sufficient consideration of these complex dependencies, traditional deep reinforcement learning exhibits certain limitations when dealing with complex dependencies between tasks. In particular, when faced with large-scale and high-dimensional tasks, it struggles to effectively capture global dependencies, resulting in weak generalization and scalability, and in turn, inaccurate task allocation.

[0071] Furthermore, in the DQN (Deep Q-Network), a traditional privacy-preserving task allocation approach, the calculation of Q values ​​and action selection are performed by the same network. This single network struggles to effectively distinguish between state and action values, and is prone to bias when updating Q values ​​via the Bellman equation. This can lead to overestimation of Q values, which in turn causes learning instability and the generation of suboptimal policies. Furthermore, standard experience replay aims to improve training stability by randomly sampling past experiences from an experience pool to break correlations between samples. However, this approach also has potential issues, such as the possibility of overlooking key experiences, especially in the early stages of training when the agent's understanding of the environment is incomplete, making it easy to miss important opportunities for policy adjustments. Furthermore, the traditional ε-greedy strategy can achieve a certain degree of exploration, but due to the static setting of the ε value, it often fails to effectively guide the agent to explore more effectively, resulting in low learning efficiency.

[0072] In order to solve the above-mentioned defects, an embodiment of the present application provides a task assignment privacy protection method. The specific implementation method of the present application is further described below with reference to the accompanying drawings.

[0073] refer to Figure 3 FIG2 is a flowchart of a task assignment privacy protection method provided in accordance with the first aspect of an embodiment of the present application. It should be noted that the task assignment privacy protection method is applied to a perception platform, which is a crowd-sensing service provider responsible for receiving perception task requests from task requesters, assigning appropriate tasks to users (i.e., workers), collecting and storing the perception data submitted by the users, and performing processes such as truth discovery.

[0074] exist Figure 3 In the task assignment privacy protection method, the task assignment privacy protection method includes:

[0075] S1. Receive a first state graph of the current environment, wherein the first state graph is constructed by at least one user attribute information and at least one task information, and each user attribute information is obtained by noise processing using the Laplace mechanism to satisfy differential privacy.

[0076] The current environment can be a physical device, a virtual space, a software application, an abstract concept space, or a combination thereof, depending on the application scenario of the intelligent agent.

[0077] Due to the use of the first state graph, the embodiment of the present application solves the online task allocation problem as a graph-based constrained optimization problem.

[0078] The first state diagram can be expressed as G t =(U t ,V t ,E t )=(M t ,E t ), where G t is the first state diagram; U t is a set of user nodes, that is, a set of all user attribute information (it can be understood that a user node is a piece of user attribute information); V t It is a set of task nodes, that is, a set of all task information (it can be understood that a task node is a piece of task information); E t is the set of edges between user nodes and task nodes; M t M is the set of user nodes and task nodes in time slot t. t The original features of each node (element) can be represented by a vector m, m = {x, y, f u,t ,c u,x u,t ,b v ,t v ,T v,t ,F v,a ,F v,f ,ms}, where x and y are the target nodes (i.e., M t The position of any node in the u When x and y are the starting position information of the user node, when the target node is the task node m v When x and y are the task positions of the task node; f u,t is the maximum travel capability of the user node in time slot t; c u is the unit travel cost of the user node; x u,t is the maximum task processing capability of the user node in time slot t; b v is the task budget of the task node; t v is the task expiration time of the task node; T v,t is the time required to complete the task of the task node in time slot t; F v,a Is the allocation flag of the task node; F v,f Is the completion flag of the task node; ms is used to distinguish M t There are two types of nodes (i.e. user nodes and task nodes). When ms=1, the target node is the user node m u , when ms=0, the target node is task node m v For user node m u , initially defined as completing a b v , t v 、T v,t 、F v,a and F v,f All virtual tasks are 0; for task node m v , then define f at the initial u,t 、c u and x u,t The user attributes are all 0. In other implementations, the vector m also includes a node ID.

[0079] For E t , this embodiment of the application defines two types of edges. The first type of edge is the edge between the user (node) and the task (node). u,v (t); The second type of edge is the edge between users (nodes) and tasks (nodes) u1,u2 (t) and e v1,v2 (t).

[0080] Furthermore, regarding the first type of edge e u,v(t), the weight of each first-type edge is calculated according to the Manhattan distance between the two ends of the edge at time slot t. The specific calculation formula is:

[0081] L1(u,v)=|l u,t -l v,t | x +|l u,t -l v,t | y ;

[0082] e u,v (t) = L1(u,v);

[0083] Wherein, L1(u,v) is the Manhattan distance between the two ends of the edge at time slot t, u is the target user attribute information, and the target user attribute information is any one of all user attribute information; v is the target task information, and the target task information is any one of all task information; l u,t is the starting position information of the target user attribute information in time slot t; v,t is the task position of the target task information in time slot t; |l u,t -l v,t | x In the x-axis direction u,t and l v,t The distance between |l u,t -l v,t | y In the y-axis direction l u,t and l v,t The distance between u,v (t) is the weight of the edge between the target user attribute information and the target task information in time slot t.

[0084] Furthermore, the second type of edge is the edge between users and tasks. and The weight of each second-type edge is calculated based on the Manhattan distance between the two ends of the edge at time slot t and the cosine similarity of its features. The specific calculation formula is:

[0085]

[0086] Among them, u1 is the target user attribute information, u2 is a user attribute information other than the target user attribute information in all user attribute information, The target user attribute information is in M t The corresponding node in is a user attribute information in all user attribute information except the target user attribute information in M tThe corresponding node in , v1 is the target task information, v2 is a task information in all task information except the target task information, The target task information is in M t The corresponding node in For a task information other than the target task information in all task information, t The corresponding node in, cos() is the cosine similarity, is the edge between user attribute information (it can also be understood as the edge between users). is the edge between task information (it can also be understood as the edge between tasks). is the starting position information of u1 in time slot t, is the starting position information of u2 in time slot t, is the task position of v1 in time slot t, is the task position of v2 in time slot t, L1(u1,u2) is the Manhattan distance between the two ends of the edge within the user attribute information, L1(v1,v2) is the Manhattan distance between the two ends of the edge within the task information, In the x-axis direction and The distance between In the y-axis direction and The distance between In the x-axis direction and The distance between In the y-axis direction and The distance between them.

[0087] It should also be noted that, in the embodiment of the present application, one user corresponds to one piece of user attribute information, and one task corresponds to one piece of task information.

[0088] The user attribute information includes starting location information, maximum travel capacity, unit travel cost, and maximum task processing capacity. To protect the privacy of each user (i.e., worker) and prevent attackers from obtaining and abusing each user's real location, this embodiment of the application uses a local differential privacy method based on the Laplace mechanism to protect the user's location. Therefore, the user location received by the sensing platform is the location information after perturbation processing.

[0089] More specifically, the user's location can be represented by coordinates, which are composed of a longitude coordinate component and a latitude coordinate component. Since coordinates are numerical data, the embodiment of the present application uses the Laplace mechanism to add random variables that satisfy the Laplace distribution to the longitude coordinate component and the latitude coordinate component respectively, so as to perturb the longitude coordinate component and the latitude coordinate component respectively, thereby achieving the purpose of satisfying the user's location privacy. The random variables (i.e., random noise values) corresponding to the longitude coordinate component and the latitude coordinate component satisfy the mean of 0 and the variance of 2b. 2 Laplace(b) distribution. For Laplace(b) distribution, its probability density function is Where b is the noise scaling factor, Δf is the sensitivity, ε is the privacy budget, x is the random variable, and e is the exponential function with the natural constant e as the base, which is used to describe the probability density when the random variable takes a certain value.

[0090] In the embodiment of the present application, Laplace noise is added to the real coordinates of the target user (that is, any one of all users), and the calculation formula for the position coordinates after the noise is obtained is: Among them, l ′ u,t is the position coordinate after adding noise; x j ′ is the horizontal coordinate data of the position coordinate after adding noise; j ′ is the vertical coordinate data of the position coordinate after noise addition; j is the longitude coordinate component of the real coordinates of the target user; j is the dimensional coordinate component of the target user's real coordinate; Δf is the sensitivity; ε is the privacy budget.

[0091] The position coordinates after adding noise can be understood as confused positions.

[0092] Each of the task information includes a task location, a task budget, a task expiration time, a task completion time, an allocation flag, and a completion flag. The allocation flag and the completion flag are both initialized to 0.

[0093] After the embodiment of the present application obtains the confused position of the user and the true position of the perception task (that is, the task position), the perception platform needs to assign the perception task to the appropriate user online, so as to optimize the objective function, that is, minimize the moving distance of the perception user, so as to maximize the benefits of the perception platform.

[0094] Furthermore, the allocation cost of the target task information (i.e., any one of the task information) allocated by the perception platform to the target user corresponding to the target user attribute information (i.e., any one of the at least one user attribute information) can be calculated by combining the starting position information, the task position, and the unit travel cost in the current time slot t. Specifically, the calculation formula of the allocation cost is: C u,v =c u ||l u,t -l v ||; Among them, C u,v is the allocation cost, c u is the unit travel cost of the target user attribute information, ||l u,t -l v || is the Manhattan distance between the starting location information of the target user attribute information and the task location of the target task information at the current time slot t, l u,t is the starting position information of the target user attribute information at the current time slot t, l v is the task position of the target task information, u is the target user attribute information, and v is the target task information (can also be expressed as target task).

[0095] Furthermore, in each time slot, in order to match the optimal solution that ultimately brings more benefits to the platform under the limited walking ability of each user, in the embodiment of the present application, the calculation formula for the platform benefit brought by assigning the target task information to the target user is: B u,v =b v -C u,v Among them, B u,v is the platform benefit brought by assigning the target task information to the target user, b v is the task budget of the target task information, C u,v is the allocation cost.

[0096] Furthermore, the calculation formula for the platform benefits brought by all the allocation results of the current time slot is: B t =∑ u∈J ∑ v∈V A u,v B u,v Among them, B t is the platform revenue brought by all allocation results; u is the target user attribute information; J is all user attribute information; v is the target task information; V is all target task information; a u,v Whether to assign the target task information v to the target user attribute information u (ie, the target user); A u,v∈{0,1}, when the target user attribute information u is assigned the target task information v, a u,v is 1, when the target task information v is not assigned to the target user attribute information u, A u,v is 0; B u,v It is the platform benefit brought by allocating the target task information to the target user.

[0097] For the perception platform, there are two ways to increase platform revenue: one is to allow users to perform tasks with higher task rewards as much as possible to obtain more task rewards; the other is to select users who are closest to the task or have a low unit travel cost to perform the task as much as possible to reduce distribution costs.

[0098] Furthermore, the reward function corresponding to the reward is expressed as:

[0099]

[0100] Among them, R(s t ,a t ) is the reward function, that is, in s t Execute a t The reward value obtained when s t is the second state diagram; a t is the first action; d p is the idle penalty, i.e., the minimum Manhattan distance among the user's available tasks; c u is the unit travel cost; τ is a fixed time penalty, τ<0; AS means no task is assigned to the user (ie, the worker); b v is the task budget; d is the Manhattan distance from the user to the task location.

[0101] In the embodiment of the present application, the second state diagram can be represented as s t or G t .

[0102] This embodiment of the present application incorporates the lowest cost among the user's available tasks as a personalized inactivity penalty into the reward function, encouraging users to choose between tasks and avoid inactivity. Furthermore, this setting makes users more inclined to choose cost-effective tasks rather than stagnate, thereby enhancing dynamism and enabling adjustments based on actual conditions. When a user does not move, the penalty imposed reflects the missed optimal task choice, thus prompting the user to take action.

[0103] It should also be understood that the reward function is one of the five core elements (or quintuples) in the Markov decision process. In the Markov decision process, the quintuples are the state space S, the action space A, the state transition function P, the reward function R, and the discount factor (or discount coefficient) γ, γ∈[0,1], which is used to weigh the importance of long-term rewards and immediate rewards. The Markov decision process can transform the problem to be solved into a reinforcement learning problem.

[0104] For the state space, the embodiment of the present application uses the second state diagram G t =(U t ,V t ,E t )=(M t ,E t ) as state input. E t The internal weights represent the relationships between users, users and tasks, and tasks and tasks. t The definition of internal weights is also different.

[0105] The action space is a vector ux3, which indicates whether each user is assigned a feasible user-task pair (user number, task number, Manhattan distance) or whether the user remains in place without any task assignment. Therefore, the action space A only includes feasible user-task pairs and user-specific stop actions. "Feasible" means that after selecting a user-task pair, the following constraints are still met:

[0106]

[0107] in, is a universal quantization symbol; u is the target user attribute information (i.e., any one user attribute information among all user attribute information); U is all user attribute information (one user corresponds to one user attribute information); v is the target task information (i.e., any one task information among all task information); V is all task information; B is the total revenue of the perception platform; T is the total duration, T = {1,…,t,…,T}; t is the current time slot t; B t Platform benefits from all distribution results; A u,v b) whether to assign the target task information v to the target user attribute information u (ie, the target user); v The task budget for the target task information; c u The unit travel cost of the target user attribute information; u,t is the starting position information of the target user attribute information at the current time slot t; v is the task location of the target task information;u,t+1 is the maximum task processing capacity of the target user attribute information in time slot t+1; x u,0 is the maximum task processing capability of the target user attribute information in time slot 0; f u,t+1 is the maximum travel capability of the target user attribute information in time slot t+1; f u,t is the maximum travel capability of the target user attribute information in time slot t; f u,0 is the maximum traveling capability of the target user attribute information in time slot 0; Γ v is the completion time of the assigned task; t v is the task expiration time; V A is the set of assigned tasks; T v,t+1 is the time required to complete the task in time slot t+1; T v,t is the time required to complete the task in time slot t; T v,0 is the time required to complete the task in time slot 0; V u′ is the task set of user u′; u′ is another user different from u; V u is the task set of user u; r u,t+1 is the reward corresponding to the target user attribute information in time slot t+1; r u,0 is the reward corresponding to the target user attribute information in time slot 0.

[0108] S2. Use the Noisy D3QN-G algorithm based on the graph attention mechanism to process the first state graph to obtain the task allocation result.

[0109] The Noisy D3QN-G algorithm is an algorithm based on the graph attention mechanism. Since the mobile crowd intelligence perception system (i.e., the perception platform) is dynamic, the perception users move randomly, the perception tasks arrive at the platform in real time, and the user and task information cannot be predicted in advance, the process of solving the online task allocation method by the Noisy D3QN-G algorithm can be referred to. Figure 4 shown.

[0110] The Noisy D3QN-G algorithm incorporates advanced technologies such as dual deep Q-networks, adversarial deep Q-networks, prioritized experience replay, and noisy networks. It mitigates issues such as overestimation of Q values, low exploration efficiency, unstable training, and the tendency to get stuck in local optima. This enables the agent (i.e., the perception platform) to more efficiently explore its environment and achieve more robust learning in high-dimensional and dynamic environments.

[0111] Specifically, the Dual Deep Q Network reduces overestimation of Q values ​​by introducing independent action and target networks, thereby improving the stability and accuracy of learning. The Adversarial Deep Q Network, by separating the state-value function from the action advantage, can better capture the potential value of the state and improve the model's performance in complex environments. Prioritized Experience Replay accelerates the training process and improves exploration efficiency by weighting the experience and prioritizing important experiences for learning. The introduction of the Noise Network further enhances the agent's exploration capabilities, while ensuring the diversity and randomness of exploration, reducing sensitivity to hyperparameters, enabling the agent to more efficiently optimize strategies in high-dimensional, dynamic, and complex environments. Furthermore, the Noise Network improves the robustness of the learning process by dynamically adjusting the noise, avoiding the problem of over-reliance on local information.

[0112] In the above implementation, the present embodiment effectively models the relationships between tasks and users through a graph structure. Simultaneously, the Noisy D3QN-G algorithm is introduced to fully capture the characteristics, potential interactions, and global dependencies between tasks and users, thereby providing a more accurate decision-making basis for online task allocation. Compared with the prior art, the present embodiment can fully consider the various nonlinearities and spatiotemporal dependencies between tasks and users, thereby improving the accuracy of task allocation.

[0113] In addition, the embodiment of the present application uses the Laplace mechanism to perform noise processing on user attribute information and disturbs the user's location information, effectively ensuring the security of the user's privacy information, improving the enthusiasm of user participation to a certain extent, and promoting the long-term development of the system.

[0114] Furthermore, the Noisy D3QN algorithm, while fully considering the dynamic nature of the system (i.e., the perception platform) and the unknown factors (unpredictable user attribute information and task information) in the system, enhances the model's ability to model complex dependencies in online task allocation, improving the system's flexibility and adaptability in responding to large-scale task and resource changes.

[0115] In some embodiments, the task assignment privacy protection method further includes a training step of the Noisy D3QN-G algorithm based on the graph attention mechanism, and the training step of the Noisy D3QN-G algorithm based on the graph attention mechanism includes:

[0116] S31. Receive the second state diagram of the historical environment.

[0117] The historical environment and the current environment should be the same environment.

[0118] The data structure and data format of the second state diagram are the same as those of the first state diagram.

[0119] S32: Input the second state diagram into the action network with the noise network in the original Noisy D3QN-G algorithm to calculate the Q value and obtain a first Q value.

[0120] It should be understood that in reinforcement learning, the trade-off between exploration and utilization is a core issue. The intelligent agent needs to explore new strategies in an unknown environment while utilizing the best known strategy. Although the traditional ε-greedy strategy can achieve exploration, it often fails to effectively guide the exploration process, resulting in low learning efficiency. The noise network introduces noise into the Q network, thereby increasing the randomness of the strategy and improving the exploration ability. This method demonstrates higher learning efficiency and strategy optimization capabilities in many complex tasks, enabling the intelligent agent to better adapt to dynamic and unknown environments. Therefore, the embodiment of the present application introduces the noise network technology to transform the deep learning network.

[0121] The noise network's architecture is characterized by its ability to dynamically generate and adjust noise, ensuring a consistent level of exploration during training. Typically, a normal distribution is used to generate noise, and the noise amplitude is gradually reduced as training progresses to balance exploration and exploitation.

[0122] Furthermore, the step S32 includes:

[0123] S321. Set a first weight, a second weight, a first bias, a second bias, and a noise factor.

[0124] S322: Generate first and second random noises corresponding to the target linear layer from a standard normal distribution using a noise factor, where the target linear layer is any linear layer in the noise network.

[0125] Furthermore, the calculation formula of the first random noise corresponding to the target linear layer is: ∈ w =σ init *N(0,1); where ∈ w is the first random noise corresponding to the target linear layer, σ init is the noise factor, N is the symbol of the standard normal distribution, 0 is the mean, and 1 is the normal distribution with a standard deviation of 1.

[0126] The calculation formula of the second random noise corresponding to the target linear layer is: b =σ init *N(0,1); where ∈ b is the second random noise corresponding to the target linear layer, σ initis the noise factor, N is the symbol of the standard normal distribution, 0 is the mean, and 1 is the normal distribution with a standard deviation of 1.

[0127] S323: Obtain a third weight and a third bias of the target linear layer by calculation based on the first weight, the second weight, the first bias, the second bias, the first random noise, and the second random noise.

[0128] Furthermore, the calculation formula of the third weight is w=μ w +σ w ⊙∈ w ; Wherein, w is the third weight, μ w is the first weight, σ w is the second weight, ∈ w is the first random noise corresponding to the target linear layer.

[0129] The calculation formula of the third bias is b=μ b +σ b ⊙∈ b ; Wherein, b is the third bias, μ b is the first bias, σ b For the second bias, ∈ b is the second random noise corresponding to the target linear layer.

[0130] S324: Perform forward propagation using the third weight and the third bias to obtain a noisy output corresponding to the target linear layer.

[0131] S325 . After obtaining the noisy outputs corresponding to all linear layers, perform Q value calculation on the second state graph to obtain the first Q value.

[0132] Furthermore, after the step S325, the step S32 further includes:

[0133] S326: Update the Q-value function of the noise network using the third weight and the third bias.

[0134] The S326 may be understood as performing Q learning update using the third weight and the third bias.

[0135] S327 . Update the third network parameter and the fourth network parameter of the noise network through back propagation.

[0136] The third network parameter is a μ parameter, and the fourth network parameter is a σ parameter.

[0137] Furthermore, the step of performing Q value calculation on the second state diagram to obtain the first Q value includes:

[0138] S3251. Embed the original features of all nodes of the second state graph into the user vector and the task vector.

[0139] The implementation of step S3251 can facilitate the input of original features of all nodes of the second state diagram into the neural network.

[0140] Furthermore, the calculation formula of the user vector is: u,t =ELU(W u ·(M t ·ms)+σ u ); where h u,t is the user vector; ELU is the exponential linear unit activation function; W u is the embedding layer weight matrix of the user node; M t is the original feature of all nodes in the state diagram (that is, the set of user nodes and task nodes in time slot t); ms is used to distinguish M t There are two types of nodes (i.e. user nodes and task nodes). When ms=1, the target node is the user node m u , when ms=0, the target node is task node m v ; σ u is the embedding layer bias term of the user node.

[0141] Furthermore, the calculation formula of the task vector is: v,t =ELU(W v ·(M t (1-ms))+σ v ); where h v,t is the task vector; ELU, M t The meanings of and ms are the same as those in the user vector formula; W v is the embedding layer weight matrix of the task node; σ v is the embedding layer bias of the task node.

[0142] S3252. Use a first graph attention mechanism to process the user vector and the set of edges between the user node and the task node in the second state graph to obtain a user feature representation; use a second graph attention mechanism to process the task vector and the set of edges between the user node and the task node in the second state graph to obtain a task feature representation.

[0143] The structures of the first and second graph attention mechanisms are exactly the same. Both the first and second graph attention mechanisms are GAT (GAT is a neural network architecture for processing graph-structured data).

[0144] The user feature representation can be expressed as h u,t ′. The task feature representation can be expressed as h v,t ′.

[0145] The update of user node u in time slot t in the second state diagram can be expressed as the following process:

[0146] First, calculate the first attention coefficient of user node u to neighbor node j. The calculation formula of the first attention coefficient is: uj =LeakyReLU(a T [Wh u,t ||Wh j,t ]); where LeakyReLU() is the rectified linear unit function; a T is the learned weight vector; W is the weight matrix for linear transformation; h u,t is the user vector corresponding to the user node u in time slot t; h j,t is the user vector corresponding to the neighbor node j in time slot t; || is the connection operation.

[0147] Secondly, the first attention coefficient is normalized. The specific normalization calculation formula is:

[0148]

[0149] Among them, α uj is the normalized attention weight of user node u to neighbor node j; is the neighbor set of user node u; k1 is The k1th neighbor node in e uk is the attention coefficient of user node u to neighbor node k1.

[0150] Finally, aggregate the neighbor features. The calculation formula for aggregation is as follows:

[0151]

[0152] Among them, h u,t ′ is the updated feature representation of user node u, that is, the user feature representation.

[0153] The update of the task node v in the time slot t in the second state diagram can be expressed as the following process:

[0154] First, calculate the second attention coefficient of the task node v to the neighbor node q. The calculation formula of the second attention coefficient is: vq =LeakyReLU(a T [Wh v,t ||Wh 1,t]); where LeakyReLU() is the rectified linear unit function; a T is the learned weight vector; W is the weight matrix for linear transformation; h v,t is the task vector corresponding to the task node v in time slot t; h q,t is the task vector corresponding to the neighbor node q in time slot t; || is the connection operation.

[0155] Secondly, the second attention coefficient is normalized. The specific normalization calculation formula is:

[0156]

[0157] Among them, α vq is the normalized attention weight of the task node v to the neighbor node q; is the neighbor set of task node v; k2 is The k2th neighbor node in e vk2 is the attention coefficient of the task node v to the neighbor node k2.

[0158] Finally, aggregate the neighbor features. The calculation formula for aggregation is as follows:

[0159]

[0160] Among them, h v,t ′ is the updated feature representation of the task node v, that is, the task feature representation.

[0161] S3253. Calculate the attention factor using a preset algorithm.

[0162] Specifically, the calculation formula of S3253 is:

[0163] h js =ELU(W hjs ·x j +σ hjs );

[0164] h′ js =Tanh(W′ hjs ·h js +σ′ hjs );

[0165] a su =Softmax(W asu ·h′ js +σ asu );

[0166] Among them, a su is the attention factor, x j ={f j,t ,cj ,x j,t};f j,t is the maximum traveling capacity; c j is the unit travel cost; x j,t is the maximum task processing capacity; j is equivalent to the jth user attribute information; t is the time slot t; W hjs 、W h ′ js and W asu are weight matrices used for linear transformation; σ hjs , σ h ′ js and σ asu Both are bias terms; Tanh() is the activation function; Softmax() is the activation function.

[0167] In this embodiment of the application, the attention factor a is calculated su To measure the status s of different user nodes t importance.

[0168] S3254. Calculate and obtain a state and a second action based on the attention factor, the user feature representation, and the task feature representation.

[0169] Specifically, the calculation formula of the state is: t =ELU(W st ·((a su ·h u,t ′)||(h v,t ′·F v,f ))+σ st ); The calculation formula for the second action is: a t =ELU(W at ·(h u,t ′||h v,t ′)+σ at ); where W st and W at are weight matrices used for linear transformation; F v,f Is the completion flag; σ st and σ at All are bias terms.

[0170] S3255: Calculate and obtain the first Q value based on the state and the second action.

[0171] Specifically, the calculation formula of step S3255 is:

[0172] Q value =W qt ·ELU(W′ qt ·(st ||a t )+σ′ qt )+σ qt ;

[0173] Among them, Q value is the first Q value, W qt and W′ qt are weight matrices used for linear transformation; σ′ qt and σ qt The first Q value is a Q value matrix.

[0174] Through the above operations, the embodiment of the present application can effectively learn from the features of users and task nodes and calculate the Q value of each possible action. At the same time, the introduction of the attention mechanism enables the network to dynamically adjust the contribution of different features to the final Q value.

[0175] Furthermore, after the step S3255, the step of performing Q value calculation on the second state graph to obtain the first Q value further includes:

[0176] S3256. Use the value function of the adversarial deep Q network in the original Noisy D3QN-G algorithm to estimate the overall value of the state.

[0177] S3257. Use the action advantage function of the adversarial deep Q network in the original Noisy D3QN-G algorithm to estimate the action advantage of taking the second action in the state.

[0178] S3258. After combining the overall value and the action advantage, a final Q-value estimate is generated.

[0179] The calculation formula for the final Q value estimation is as follows:

[0180]

[0181] in, is the mean of all action advantages, so that when estimating any state s and selected action a, the mean value of the advantage function is guaranteed to be zero. The Q function output by the network is generated by combining the state value and action advantage and subtracting the mean value of the action advantage.

[0182] This approach effectively reduces meaningless offsets, ensures that the advantage estimates of different actions do not affect each other, and prevents the Q value from being disturbed by excessive offsets, which helps optimize the network more stably.

[0183] Unlike traditional deep reinforcement learning methods, the graph attention mechanism assigns different weights to each node and its neighboring nodes, enabling the model to learn autonomously and extract key features from complex task dependencies. This not only enhances the model's ability to model complex dependencies in online task allocation, but also improves the system's flexibility and adaptability when dealing with large-scale task and resource changes. This enables the embodiments of the present application to have stronger generalization and scalability, and can efficiently handle changes in the number of dynamic tasks and resources, thereby quickly approaching the global optimal allocation strategy.

[0184] It should also be noted that in the reinforcement learning model of the present embodiment (i.e., the Noisy D3QN-G algorithm), states are represented by state graphs, which results in significant differences in the graph structure between different problem instances. Traditional neural networks have fixed-size inputs and outputs, while dynamically changing state graph inputs are incompatible with traditional neural networks, posing a major challenge in solving graph-based constraint programming problems.

[0185] To address this issue, the present invention introduces a Graph Attention Network (GAT) to handle changing state graph inputs. GAT dynamically adjusts attention weights based on the relationships between nodes, enabling the network to effectively capture graph structure information. This combination not only solves the problem of input incompatibility but also improves the model's performance when processing complex graph structures.

[0186] The Noisy D3QN algorithm in the embodiment of the present application is the Noisy D3QN algorithm of MGDRL.

[0187] refer to Figure 5 As shown, the main structure of the MGDRL framework in this embodiment of the application has four main components: the embedding layer, the graph attention mechanism layer, the Q-learning layer (also known as the Q-value calculation layer), and the Dueling layer. Traditional models only input feature information related to one available user-task pair at a time and then obtain the corresponding Q-value. However, this embodiment of the application directly inputs the entire state graph into the network and then obtains a global Q-value matrix.

[0188] More specifically, in the embedding layer, step S3251 is implemented.

[0189] In the graph attention mechanism layer, implement step S3252.

[0190] In the Q-learning layer, steps S3253-S3255 are implemented.

[0191] In the Dueling layer, steps S3256-S3258 are implemented. In the Dueling layer, two fully connected layers are used to represent state value and action advantage. This structure enables the separate estimation of state value and the advantage of each action, thereby improving decision-making quality during online task allocation. It should be understood that the core idea of ​​the Dueling Deep Q-Network (Dueling DQN) is to independently learn state value and action advantage and combine them in a certain way to form a Q-value function. In a traditional DQN, the network directly outputs the Q-value Q(s, a) corresponding to each action. In the Dueling Deep Q-Network, the network is divided into two branches: the state value V(s) and the action advantage A(s, a). The state value estimates the overall value of state s, independent of the action; the action advantage estimates the advantage of taking action a in state s, reflecting the superiority of that action over other actions.

[0192] S33: Select the first action with the largest first Q value.

[0193] The calculation formula of step S33 is: in, is the first action; Q eval () is the action network; is the first Q value; θ is the first network parameter of the action network; a is any feasible action; is the current state, that is, the second state diagram.

[0194] S34. After sending the first action to the historical environment, receive the next state diagram and reward fed back by the historical environment based on the first action.

[0195] S35: Store the second state diagram, the first action, the reward, and the next state diagram as an experience in an experience replay area, and prioritize all experiences in the experience replay area.

[0196] In standard experience replay, the agent randomly samples past experiences from the experience pool for learning. This somewhat breaks down the correlation between samples, but may cause some key experiences to be overlooked. Prioritized experience replay, on the other hand, prioritizes experiences, giving those that contribute more to the learning process a higher probability of sampling.

[0197] Furthermore, the step of prioritizing all experiences in the experience playback area includes:

[0198] S351. Calculate the TD (Temporal Difference) error of the target experience, where the target experience is any experience in the experience playback area.

[0199] Furthermore, the calculation formula of the target experience TD error is:

[0200] δ i =r t +γarg max a′ Q(s t+1 , a t+1 ;ω)-Q(s t , a t ;ω);

[0201] Among them, δ i is the TD error of the target experience, that is, the TD error of the i-th experience; r t is the reward; γ is the discount factor, which is used to reduce the impact of future rewards on current decisions and ensure the finiteness of long-term rewards; a is Q(s t+1 , a t+1 ;ω) the maximum action achieved; Q(s t+1 , a t+1 ω) is the agent (i.e., the perception platform) in time slot t+1 to s t+1 and a t+1 Expected return; s t+1 is the next state; a t+1 is the next action; ω is the parameter of the Q value function; Q(s t , a t ;ω) is the agent’s response to s in time slot t t and a t Expected return; s t is the second state diagram; a t This is the first action.

[0202] The TD error is used to measure the importance of experience. The larger the TD error, the greater the impact of the target experience on the Q value update, and thus the higher the priority of the target experience.

[0203] S352: Calculate the priority of the target experience based on the TD error.

[0204] Furthermore, the calculation formula of S352 is:

[0205] p i =|δ i |+∈ p ;

[0206] Among them, p iis the priority of the target experience, that is, the priority of the i-th experience; δ i is the TD error of the target experience; ∈ p is a positive value to avoid the situation where the probability is zero. It should be understood that ∈ p It is a very small positive value.

[0207] S353. Prioritize all experiences in the experience playback area based on the priorities of all experiences.

[0208] S36. When the number of all experiences in the experience replay area reaches or exceeds the requirement of a preset batch of training data, the experience replay area is randomly sampled with priority to obtain experience samples, and the experience samples are input into the action network for Q value calculation to obtain a second Q value, and the experience samples are input into the target network in the original Noisy D3QN-G algorithm for Q value calculation to obtain a third Q value.

[0209] The preset batch training data is 64 batches of training data. In other embodiments, it can also be 128 batches of training data or 256 batches of training data. The specific setting can be made by those skilled in the art according to actual needs.

[0210] Furthermore, the step of performing random sampling of the priority on the experience playback area to obtain experience samples includes:

[0211] S361. Calculate the probability of the target experience based on the priority.

[0212] Furthermore, the calculation formula of step S361 is:

[0213]

[0214] Where P(i) is the probability of the target experience, that is, the probability of the i-th experience; p i is the priority of the target experience; α p is a hyperparameter used to control the degree of priority, α p The larger the value of , the more biased towards high priority samples. p =0, the step S361 is equivalent to simple random sampling; k is the priority of the kth experience.

[0215] It should be understood that when the embodiment of the present application extracts samples from the experience pool, sampling is performed according to the priority of the experience, and the more important the experience, the greater the probability of being selected.

[0216] S362: Sampling from the experience playback area based on the probability to obtain the experience sample.

[0217] Furthermore, after the step S362, the step of performing random sampling of the experience playback area according to priority to obtain experience samples further includes:

[0218] S363. Set the first coefficient.

[0219] S364. Calculate an importance sampling weight based on the first coefficient, the total number of all experiences in the experience playback area, and the probability of the target experience.

[0220] In order to reduce bias and ensure unbiased estimation, the embodiment of the present application modifies the priority experience using importance sampling weights. These weights are balanced by the first coefficient to reduce the bias caused by priority sampling.

[0221] Furthermore, the calculation formula of step S364 is:

[0222]

[0223] Among them, w p,i is the importance sampling weight (which can also be expressed as the importance sampling weight corresponding to the i-th experience and its priority p); N is the total number of all experiences in the experience playback area (which can also be expressed as the size of the experience pool); P(i) is the probability of the target experience; β p is the first coefficient, which can also be understood as a hyperparameter.

[0224] It should be understood that as training progresses, β is usually increased gradually. p The value of β p Towards 1 to reduce the deviation.

[0225] S365: Update the Q value based on the importance sampling weight.

[0226] The calculation formula of S365 is:

[0227]

[0228] in, is the updated Q value; Q(s i ,a i ; ω) is the Q value; α p is a hyperparameter; w p,i is the importance sampling weight; δ i is the TD error of the target experience; s i It can be understood as s t ;a i Can be understood as a t ;ω is the parameter of the Q-value function.

[0229] Furthermore, the specific steps of calculating the second Q value are the same as the specific steps of calculating the first Q value, and are not repeated here.

[0230] Furthermore, the empirical sample is input into the target network in the original Noisy D3QN-G algorithm to calculate the Q value, and the calculation formula for the third Q value step is:

[0231]

[0232] in, is the third Q value; r t is the reward; γ is a discount factor used to reduce the impact of future rewards on current decisions and ensure the finiteness of long-term rewards; Q target () is the target network; s t+1 is the next state; a t is the first action; θ - is a second network parameter of the target network.

[0233] Furthermore, after the step S36, the training step of the Noisy D3QN-G algorithm based on the graph attention mechanism further includes:

[0234] S39: Update the action network based on the third Q value to obtain an updated action network.

[0235] Specifically, the calculation formula of step S39 is:

[0236]

[0237] in, is the updated action network; Q eval is the action network (before updating); α is the learning rate; is the third Q value.

[0238] S37. Calculate a loss function based on the second Q value and the third Q value.

[0239] Furthermore, the calculation formula of the loss function is:

[0240] L ω =(r t +γarg max a′ Q(s t+1 , a t+1 ;ω)-Q(s t , a t ;ω)) 2 ;

[0241] Among them, L ω is the loss function; w t is the reward; γ is the discount factor, which is used to reduce the impact of future rewards on current decisions and ensure the finiteness of long-term rewards; a is Q(s t+1 , a t+1 ;ω) the maximum action achieved; Q(s t+1 , a t+1 ω) is the agent (i.e., the perception platform) in time slot t+1 to s t+1 and a t+1 The expected return (also known as the third Q value); s t+1 is the next state; a t+1 is the next action; ω is the parameter of the Q value function; Q(s t , a t ;ω) is the agent’s response to s in time slot t t and a t The expected return (also known as the second Q value); s t is the second state diagram; a t This is the first action.

[0242] In the embodiment of the present application, the loss function of the action network is defined as the variance between the target value and the predicted value. The target value can be understood as Q(s t+1 , a t+1 ;ω) The predicted value can be understood as Q(s t ,a t ;ω) The target value can be understood as a predicted task allocation result.

[0243] Through the above implementation, the embodiment of the present application can reduce the unstable effect caused by model noise, and reduce the excessive Q-value estimation deviation, making the estimated Q value more reliable and improving the overall performance.

[0244] S38. Adjust the first network parameters of the action network based on the loss function, and use a soft update strategy to update the second network parameters of the target network to obtain the Noisy D3QN-G algorithm based on the graph attention mechanism.

[0245] Furthermore, the step of updating the second network parameter of the target network using the soft update strategy includes:

[0246] S381. Calculate a fifth network parameter based on the first network parameter of the action network.

[0247] Furthermore, the calculation formula of step S381 is: in, is the fifth network parameter; τ is a hyperparameter in the range of (0, 1), which is usually small (e.g., 0.001) and is used to control the update speed; θ is the first network parameter of the action network; θ - is a second network parameter of the target network.

[0248] It should be understood that the embodiment of the present application uses a soft update strategy to update the target network, and only adjusts a small part of the weight of the target network each time, thereby avoiding large fluctuations caused by a one-time copy action network parameter.

[0249] The advantage of using a soft update strategy in the embodiments of the present application is that it enables smoother parameter updates, helps maintain synchronization between the action network and the target network over the long term, reduces excessive estimation errors, and thus strikes a balance between accuracy and stability. In this way, target drift is reduced, improving the learning efficiency of the embodiments of the present application.

[0250] S382: Replace the second network parameter of the target network with the fifth network parameter to update the target network.

[0251] In some implementations, before step S1, the task assignment privacy protection method further includes:

[0252] Initialize the action network and the target network.

[0253] refer to Figure 6 As shown, it is a principle block diagram of a task allocation privacy protection device provided by the second aspect of the embodiment of the present application. Figure 6 In the embodiment, the task assignment privacy protection device 100 includes:

[0254] A receiving module 101 is configured to receive a first state graph of a current environment, wherein the first state graph is constructed from at least one user attribute information and at least one task information, and each user attribute information is differentially private information obtained by performing noise processing using a Laplace mechanism;

[0255] The processing module 102 is used to process the first state graph using the Noisy D3QN-G algorithm based on the graph attention mechanism to obtain a task allocation result.

[0256] The third aspect of the embodiment of the present application provides a terminal device, the principle block diagram of the terminal device can be as follows: Figure 7As shown. The terminal device includes a processor, a memory, a network interface, a display screen and a temperature sensor connected via a system bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a task allocation privacy protection method is implemented. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor is pre-set inside the terminal device to detect the operating temperature of the internal device.

[0257] Those skilled in the art will understand that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal device to which the solution of the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0258] In some embodiments, embodiments of the present application provide a terminal device comprising a processor and a memory, the memory being configured to store a computer program, the processor being configured to call and execute the computer program stored in the memory to perform the steps of the task assignment privacy protection method provided in the first aspect of the embodiments of the present application. A fourth aspect of embodiments of the present application provides a computer-readable storage medium, the computer-readable storage medium being configured to store a computer program that causes a computer to perform the steps of the task assignment privacy protection method provided in the first aspect of the embodiments of the present application.

[0259] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0260] The technical features of the above embodiments can be combined without changing the basic principles of this application. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0261] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of patent protection for the present application shall be determined by the appended claims.

Claims

1. A task assignment privacy protection method, characterized in that: include: Receive a first state graph of the current environment, wherein the first state graph is constructed by at least one user attribute information and at least one task information, and each user attribute information is differentially private information obtained by performing noise processing using a Laplace mechanism; The Noisy D3QN-G algorithm based on the graph attention mechanism is used to process the first state graph to obtain the task allocation result; The task assignment privacy protection method also includes a training step of the Noisy D3QN-G algorithm based on the graph attention mechanism, and the training step of the Noisy D3QN-G algorithm based on the graph attention mechanism includes: receiving a second state graph of the historical environment; inputting the second state graph into the action network with the noise network in the original Noisy D3QN-G algorithm to calculate the Q value and obtain a first Q value; selecting the first action with the largest first Q value; after sending the first action to the historical environment, receiving the next state graph and reward fed back by the historical environment according to the first action; storing the second state graph, the first action, the reward and the next state graph as an experience in the experience replay area, and prioritizing all experiences in the experience replay area; when the number of all experiences in the experience replay area reaches or exceeds the requirement of a preset batch of training data, performing priority random sampling on the experience replay area to obtain experience samples, and inputting the experience samples into the action network to calculate the Q value to obtain a second Q value, and inputting the experience samples into the original Noisy The target network in the D3QN-G algorithm calculates a Q value to obtain a third Q value; a loss function is calculated based on the second Q value and the third Q value; a first network parameter of the action network is adjusted based on the loss function, and a second network parameter of the target network is updated using a soft update strategy to obtain the Noisy D3QN-G algorithm based on the graph attention mechanism; The step of inputting the second state graph into the action network of the noisy network in the original Noisy D3QN-G algorithm to perform Q value calculation to obtain the first Q value includes: setting a first weight, a second weight, a first bias, a second bias, and a noise factor; using the noise factor to generate a first random noise and a second random noise corresponding to a target linear layer from a standard normal distribution, where the target linear layer is any linear layer in the noisy network; calculating a third weight and a third bias of the target linear layer based on the first weight, the second weight, the first bias, the second bias, the first random noise, and the second random noise; performing forward propagation using the third weight and the third bias to obtain a noisy output corresponding to the target linear layer; after obtaining the noisy outputs corresponding to all linear layers, performing Q value calculation on the second state graph to obtain the first Q value; The step of calculating the Q value of the second state graph to obtain the first Q value includes: embedding the original features of all nodes in the second state graph into the user vector and the task vector; using the first graph attention mechanism to process the user vector and the set of edges between the user nodes and the task nodes in the second state graph to obtain a user feature representation; using the second graph attention mechanism to process the task vector and the set of edges between the user nodes and the task nodes in the second state graph to obtain a task feature representation; using a preset algorithm to calculate the attention factor; calculating the state and the second action based on the attention factor, the user feature representation and the task feature representation; and calculating the first Q value based on the state and the second action.

2. The task assignment privacy protection method according to claim 1, characterized in that: After obtaining the noisy outputs corresponding to all linear layers, performing Q value calculation on the second state graph to obtain the first Q value, the step of inputting the second state graph into the action network with the noisy network in the original Noisy D3QN-G algorithm to perform Q value calculation to obtain the first Q value further includes: Updating a Q-value function of the noise network using the third weight and the third bias; The third network parameter and the fourth network parameter of the noise network are updated by back propagation.

3. The task assignment privacy protection method according to claim 1, characterized in that: After the step of calculating and obtaining the first Q value based on the state and the second action, the step of calculating the Q value on the second state graph to obtain the first Q value further includes: Use the value function of the adversarial deep Q network in the original Noisy D3QN-G algorithm to estimate the overall value of the state; The action advantage function of the adversarial deep Q network in the original Noisy D3QN-G algorithm is used to estimate the action advantage of taking the second action in the state; The overall value and the action advantage are combined to generate a final Q-value estimate.

4. The task assignment privacy protection method according to claim 1, characterized in that: Before the step of calculating the loss function based on the second Q value and the third Q value, the training step of the Noisy D3QN-G algorithm based on the graph attention mechanism further includes: Set the first coefficient; Calculating an importance sampling weight based on the first coefficient and the total number of all experiences in the experience playback area; The second Q value is updated based on the importance sampling weight.

5. A task assignment privacy protection device, characterized in that: include: a receiving module, configured to receive a first state graph of a current environment, wherein the first state graph is constructed by at least one user attribute information and at least one task information, and each user attribute information is differentially private information obtained by performing noise processing using a Laplace mechanism; A processing module, configured to process the first state graph using a Noisy D3QN-G algorithm based on a graph attention mechanism to obtain a task allocation result; The task assignment privacy protection device is also used to execute the training step of the Noisy D3QN-G algorithm based on the graph attention mechanism, and the training step of the Noisy D3QN-G algorithm based on the graph attention mechanism includes: receiving a second state graph of the historical environment; inputting the second state graph into the action network with the noise network in the original Noisy D3QN-G algorithm to calculate the Q value and obtain a first Q value; selecting the first action with the largest first Q value; after sending the first action to the historical environment, receiving the next state graph and reward fed back by the historical environment according to the first action; storing the second state graph, the first action, the reward and the next state graph as an experience in the experience replay area, and prioritizing all experiences in the experience replay area; when the number of all experiences in the experience replay area reaches or exceeds the requirement of a preset batch of training data, performing priority random sampling on the experience replay area to obtain experience samples, and inputting the experience samples into the action network to calculate the Q value to obtain a second Q value, and inputting the experience samples into the original Noisy The target network in the D3QN-G algorithm calculates a Q value to obtain a third Q value; a loss function is calculated based on the second Q value and the third Q value; a first network parameter of the action network is adjusted based on the loss function, and a second network parameter of the target network is updated using a soft update strategy to obtain the Noisy D3QN-G algorithm based on the graph attention mechanism; The task assignment privacy protection device is further configured to set a first weight, a second weight, a first bias, a second bias, and a noise factor; generate a first random noise and a second random noise corresponding to a target linear layer from a standard normal distribution using the noise factor, wherein the target linear layer is any linear layer in the noise network; calculate a third weight and a third bias of the target linear layer based on the first weight, the second weight, the first bias, the second bias, the first random noise, and the second random noise; perform forward propagation using the third weight and the third bias to obtain a noisy output corresponding to the target linear layer; after obtaining the noisy outputs corresponding to all linear layers, perform Q value calculation on the second state diagram to obtain the first Q value; The task allocation privacy protection device is also used to embed the original features of all nodes in the second state graph into the user vector and the task vector; use the first graph attention mechanism to process the user vector and the set of edges between the user nodes and the task nodes in the second state graph to obtain a user feature representation; use the second graph attention mechanism to process the task vector and the set of edges between the user nodes and the task nodes in the second state graph to obtain a task feature representation; use a preset algorithm to calculate the attention factor; calculate the state and the second action based on the attention factor, the user feature representation and the task feature representation; calculate the first Q value based on the state and the second action.

6. A terminal device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the steps of the task allocation privacy protection method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that Used to store a computer program, wherein the computer program enables a computer to execute the steps of the task assignment privacy protection method according to any one of claims 1 to 4.

Images