Close detection system and detection method based on federal reinforcement learning

Abstract

This invention belongs to the field of contact detection technology. Specifically, it is a close contact detection system and method based on federated reinforcement learning. The system consists of two parts: an edge user terminal and a central cloud server. The method includes two parts: close contact detection system training and close contact detection system usage. This invention ensures that, in a federated learning environment, contact data transmitted between users via Bluetooth signals is always stored on the edge user terminal, protecting user data and privacy. By utilizing a recurrent neural network to learn the correlation between Bluetooth close contact signals and distance, a close contact time inference model is learned. By inputting Bluetooth signal sequence data, the close contact time under different distance intervals is obtained. The parameter fusion method of this invention, based on reinforcement learning, uses a multi-agent reinforcement learning method to learn the weight selection of the edge user terminal during the parameter fusion process in the federated learning environment, ensuring maximum model accuracy in different training scenarios.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of contact detection, and relates to technologies such as deep learning, reinforcement learning and distributed learning. Specifically, it is a close contact detection system and method based on federated reinforcement learning, which can effectively protect user privacy while realizing personnel close contact detection. BACKGROUND

[0002] Bluetooth technology is an open global specification for wireless data and voice communication, which is a low-cost, short-range wireless personal network transmission (Wireless Personal Area Network) application. Since 2010, Bluetooth has launched a low-power version (Bluetooth Low-energy, BLE) of version 4.0, which can be in standby mode with low power consumption on mobile devices, allowing it to be connected for a long time; it does not consume as much power as Wi-Fi and does not consume as much data as 3G, and can connect with external devices at any time. Today, in the Internet era, mobile phone applications are ubiquitous, everyone has a mobile phone, and mobile phones are equipped with Bluetooth modules. We can naturally send and receive data information between users by using mobile phone Bluetooth.

[0003] The application of the Internet era has brought great convenience to people's life, but the leakage of user data also suffers great risk. For example, in April 2021, a hacker forum exposed the personal data of 533 million Facebook users in 106 countries and regions, including Facebook ID, real name, location, phone number, email and other information. More and more attention is paid to the data privacy and security of users at home and abroad, and relevant laws and regulations have been enacted. For example, the Network Security Law of China and the General Principles of Civil Law enacted in 2017 require Internet companies not to disclose or tamper with personal information they collect, and when conducting data transactions with third parties, they need to ensure that the proposed contract complies with legal data protection obligations. On May 25, 2018, the United Nations issued the General Data Protection Regulation (GDPR). GDPR aims to protect users' personal privacy and provide data security.

[0004] The close contact detection method in the prior art mostly only involves the judgment of personnel contact, and there are few diagnostic methods for close contact distance and time. SUMMARY

[0005] The application aims to provide a system and method for close contact detection using Bluetooth signal sequence data with privacy protection, which uses a federated learning environment to protect user privacy, uses the characteristics of recurrent neural networks that can learn long-term dependencies to learn the correlation between Bluetooth signal sequences and distance changes for close contact detection, and uses a multi-agent reinforcement learning method to learn the weights of the parameter fusion process in the federated learning environment to improve the accuracy of close contact detection.

[0006] The specific technical solution adopted by the application is a close contact detection system and method based on federated reinforcement learning: a federated learning environment is used to protect user privacy, and the close contact detection system is divided into edge user terminals and central service clouds. The edge user terminal acts as a participant in federated learning, uses local data collaboration to train a close contact time inference model, and uses a weight selection model to select weights in different training states to enable the locally trained close contact time inference model to better participate in global training. The central service cloud determines user contact relationships by constructing a close contact network, and constructs a training and inference user set accordingly, uses parameter fusion to achieve the purpose of protecting user privacy while sharing data, and finally uses the feedback information of parameter fusion to train a reinforcement learning model to obtain the weight selection model of the edge user terminal.

[0007] The specific implementation steps of the close contact detection method based on federated reinforcement learning disclosed in the application are as follows:

[0008] Close contact detection system training process:

[0009] Step 1: After receiving the Bluetooth signals broadcast by other edge user terminals, the edge user terminal generates a close contact identification pair and immediately uploads it to the central service cloud to construct a close contact network;

[0010] Step 2: The central service cloud selects edge user terminals participating in this round of training;

[0011] Step 3: The edge user terminal trains a close contact time inference model using local data, selects weights according to the weight selection model, and after training, uses a homomorphic encryption algorithm to encrypt the parameters of the close contact time inference model, and uploads the encrypted parameters and weight information to the central service cloud;

[0012] Step 4: After receiving the encrypted parameters and weight information uploaded by the edge user terminal, the central service cloud performs a parameter fusion process on the parameters, and distributes the fused global shared parameters to the edge user terminals participating in training;

[0013] Step 5: The edge user terminal updates the close contact time inference model using the fused global shared parameters, tests the updated close contact time inference model using the local test data set, and uploads the local test close contact time inference model accuracy to the central service cloud;

[0014] Step 6: After the central service cloud receives the close contact time inference model test accuracy of all edge user terminals participating in training, a global reward is formed, and the reinforcement learning model training module trains the weight selection model using the global reward, and distributes the trained weight selection model parameters to the edge user terminals participating in training.

[0015] Close contact detection system usage process:

[0016] Step 1: The central cloud service end determines the patient and his / her contact personnel according to the close contact network, and sends a close contact time inference instruction to the relevant edge user terminal;

[0017] Step 2: The edge user terminal determines the user relationship according to the close contact time inference instruction, obtains the close contact time inference results of different distance intervals using the close contact time inference model, and uploads the results to the central service cloud;

[0018] Step 3: The central service cloud averages the uploaded close contact time inference results according to the corresponding relationship of the close contact identification pairs of the close contact network, and distributes the averaged results to the corresponding edge user terminals.

[0019] In the above technical solution, the close contact time inference model has two parts, which are a recurrent neural network and a Softmax function. The formula representation of the recurrent neural network is:

[0020]

[0021] Where Δt is the duration of receiving Bluetooth signal data, and are the outputs of the i-1th and ith input data in the received data within Δt, is the ith input data in the received data within Δt, and θ is the parameter of the recurrent neural network. The edge user terminal stores continuously received Bluetooth signal data, which can be used to obtain the relationship between Bluetooth signal data and distance and time using the data within Δt duration. The input X Δt of the recurrent neural network can be set as the Bluetooth signal receiving data within a period of time, that is:

[0022]

[0023] Where Δt is the duration of receiving Bluetooth signal data, n is the data size of receiving Bluetooth signal data within Δt, is the i-th input data of Bluetooth signal data received in Δt time. is a two-dimensional vector, where s is the size of Bluetooth signal, and v is the time difference between receiving the i-1th data and receiving the i-th data.

[0024] Output Y of recurrent neural network Δt is the close contact probability vector, that is:

[0025]

[0026] where Δt is the duration of receiving Bluetooth signal data, and m is the number of distance intervals, is the close contact probability under distance interval i. This distance interval can be determined according to epidemic prevention requirements, for example: (0, 2)(2, 5)(5, 9)(9, +∞)(unit m), at this time m = 4, (0, 2) corresponds to distance interval 1, (2, 5) corresponds to distance interval 2, and so on.

[0027] When training the close contact time inference model, the edge user terminal can obtain the total contact time T Δt in Δt time through the received Bluetooth signal data. Δt The contact time in different distance intervals i Finally, the training label of the close contact time inference model can be obtained that is:

[0028]

[0029] When inferring the close contact time through the close contact time inference model, for the output probability value of the recurrent neural network and the case where it is not 1, the output of the recurrent neural network is controlled using the Softmax function, so that the output probability value is 1. The mathematical operation of the Softmax function is as follows:

[0030]

[0031] is the close contact probability under distance interval i obtained by the close contact time inference model. The edge user terminal can obtain the total contact time T Δt When the close contact probability of different distance intervals is output through the Softmax function , then is multiplied by the total contact time T Δt The close contact time under different distance intervals can be obtained that is:

[0032]

[0033] In the present application, in the federated learning environment, the process that edge user terminals participate in the close-time inference model training using local data is ultimately embodied by model parameter fusion. Parameter fusion is the process of weighted summation of the parameters of the close-time inference model uploaded by the edge user terminals. The size of the weight in the parameter fusion process reflects the participation degree of the edge user terminals, and the setting of the weight will affect the convergence speed of the model training and the model accuracy. The present method uses the dynamic selectivity of reinforcement learning to select the weight of the edge user terminals for the local and global training process states of different edge user terminals in the federated learning environment.

[0034] The present method specifically uses the multi-agent reinforcement learning method V-value decomposition network (VDN). Through the global reward obtained after each edge user terminal performs the weight selection action, the weight selection model of the edge user terminals can be iteratively trained.

[0035] In the present reinforcement learning environment, the following settings are made:

[0036] State: The state of each edge user terminal is composed of five parts: probe loss, global model accuracy, local data size, global data size, and local training rounds. Let denote the probe loss of edge user terminal i in the t-th training round, i.e. the loss obtained by the edge user terminal by training using local data. The probe loss represents the difference between the data of the edge user terminals; the global model accuracy M represents the model accuracy effect after the edge user terminal selects the weight for parameter fusion, and also reflects the progress of the training, with high accuracy indicating the later stage of the training; the local data size k i and the global data size K potentially reflect the quantity relationship between the local and the global, providing a basis for weighting; the local training rounds P i denote the number of local training times in one training process of the edge user terminal, and the more the training times, the more reason there is to believe that the weight should be increased; the state vector of edge user terminal i in the t-th training round The setting of the state information reflects the training state of the local and the global.

[0037] Action: In each training process, edge user terminal i selects the weight size of parameter fusion according to the state information. The action space is set to , and a weight value corresponds to a value in the range.

[0038] Reward: In order to reach the accuracy target as soon as possible, the reward signal of each iteration is set to the global model accuracy U t minus a set accuracy value u. The global model accuracy To infer the model's test accuracy for close-contact time in edge user terminal i during the t-th training round, where n is the number of edge user terminals participating in the t-th training round, maximizing the cumulative reward corresponds to finding the maximum model accuracy. The training reward r for the t-th round is... t =U t -u.

[0039] Specifically, the weight selection model for each edge user terminal uses a deep Q-network (DQN) to infer its actions, that is, it uses a deep neural network (DNN) to implement the Q function Q. θ n (s,a)=E[R t |s t n =s,a t n =a], where θ is the parameter of the DNN, This is the total global discount reward received during the t-th training round. Edge user terminal n needs to use a replay buffer to store state transition tuples in order to train locally using DQN. t n a t n s t+1 n r t >. The reinforcement learning model training module focuses on the joint Q-function. The update process is then performed. In this reinforcement learning environment, the state representation of the edge user terminals is global, so all edge user terminals share the Q-network parameters. The Q-network of the edge user terminals can be updated on the central cloud server by minimizing the loss L=E. st,at,rt,st+1 [y t -Q tot (s t ,a t )] 2 The training is performed recursively, where θ represents the parameters of the target network, and θ is periodically copied during training.

[0040] After the reinforcement learning model is trained, each edge user terminal can use the trained weights to dynamically select the corresponding weights to participate in the parameter fusion process based on the current state of the environment, thereby maximizing the global test accuracy. Simultaneously, the dynamic selectivity of reinforcement learning reduces the weights of edge user terminals with poor data quality, effectively curbing data poisoning attacks. In this method, the number of edge user terminals participating in parameter fusion each time can be different, because the state of each edge user terminal contains both global and local information, allowing each edge user terminal to select the optimal weights based on this state information.​

[0041] Consider that there are n edge user terminals in the close contact detection system training process, let u i is the i-th edge user terminal, is the t-th round of training u i The parameters of the close contact time inference model of u i The weight selected by the weight selection model in the t-th round of training is The global shared parameters after parameter fusion are θ t+1 The parameter fusion process can be represented as

[0042]

[0043] The global shared parameters θ after parameter fusion t+1 Subsequently, it will be distributed to n edge user terminals for close contact time inference model accuracy test.

[0044] The beneficial effects of the present application are:

[0045] 1. In the federated learning environment, the contact data transmitted between users through the Bluetooth signal is always saved in the edge user terminal, protecting the user data and privacy security;

[0046] 2. By using the recurrent neural network to learn the correlation between the Bluetooth close contact signal and the distance, the close contact time inference model is learned, and the close contact time in different distance intervals is obtained by inputting the Bluetooth signal sequence data.

[0047] 3. The parameter fusion method based on reinforcement learning learns the weight selection of the edge user terminal in the federated learning environment parameter fusion process by using the multi-agent reinforcement learning method, ensuring that the maximum model accuracy can be obtained in different training environments. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is the framework diagram of the present application. DETAILED DESCRIPTION

[0049] In order to deepen the understanding of the present application, the present application will be further described in detail below in combination with the drawings and examples, which are only used to explain the present application and do not limit the protection scope of the present application.

[0050] A close contact detection system based on federated reinforcement learning, in this system, it is specifically divided into two parts, which are edge user terminal and center cloud server, the structure is as Figure 1The edge user terminal is shown. There are two deep learning models in each edge user terminal, one is a close contact time inference model based on a recurrent neural network, and the other is a weight selection model in the parameter fusion process. The goal of the close contact time inference model is to use the local data of the edge user terminal to make inferences about the contact time of the user in different distance intervals. The goal of the weight selection model is to select appropriate weight values to participate in the parameter fusion process of the central cloud server after the close contact time inference model of the edge user terminal is trained once. The central service cloud includes three modules, namely the close contact network construction module, the parameter fusion module and the reinforcement learning model training module. The close contact network construction module constructs a close contact network according to the close contact identifiers uploaded by the users to determine the contact persons of the users. The parameter fusion module performs weighted summation on the model parameters uploaded by the edge user terminals to obtain new global shared parameters. The reinforcement learning model training module updates the parameters of the weight selection model of the edge user terminal using global rewards.

[0051] A close contact detection method based on federated reinforcement learning, specifically comprising the following steps:

[0052] Close contact detection system training process:

[0053] 1. The edge user terminal generates a pair of close contact identifiers after receiving the Bluetooth signals broadcast by other edge user terminals and immediately uploads them to the central service cloud to construct a close contact network;

[0054] 2. The central service cloud selects edge user terminals participating in this round of training;

[0055] 3. The edge user terminal trains the close contact time inference model using local data, selects weights according to the weight selection model, and after training, uses a homomorphic encryption algorithm to encrypt the parameters of the close contact time inference model, and uploads the encrypted parameters and weight information to the central service cloud;

[0056] 4. After the central service cloud receives the encrypted parameters and weight information uploaded by the edge user terminal, it performs a parameter fusion process on the parameters, and distributes the fused global shared parameters to the edge user terminals participating in the training;

[0057] 5. The edge user terminal updates the close contact time inference model using the fused global shared parameters, tests the updated close contact time inference model using local test data sets, and uploads the local test close contact time inference model accuracy to the central service cloud;

[0058] 6. After the central service cloud receives the close contact time inference model test accuracy of all edge user terminals participating in the training, it forms a global reward, the reinforcement learning model training module trains the weight selection model using the global reward, and distributes the trained weight selection model parameters to the edge user terminals participating in the training.

[0059] The use process of the close contact detection system is as follows:

[0060] 1. The central cloud server determines the patient and the contact personnel according to the close contact network, and sends a close contact time inference instruction to the related edge user terminal.

[0061] 2. The edge user terminal determines the user relationship according to the close contact time inference instruction, obtains the close contact time inference result of different distance intervals by using the close contact time inference model, and uploads the result to the central service cloud server.

[0062] 3. The central service cloud server averages the uploaded close contact time inference result according to the corresponding relationship of the close contact identification pair of the close contact network, and distributes the averaged result to the corresponding edge user terminal.

[0063] In the above system, the close contact time inference model has two parts, which are a recurrent neural network and a Softmax function. The formula of the recurrent neural network is as follows:

[0064]

[0065] where Δt is the duration of receiving the Bluetooth signal data, and are the outputs of the i-1th and ith input data in the received data within Δt, is the ith input data in the received data within Δt, and θ is the parameter of the recurrent neural network. The edge user terminal stores the continuously received Bluetooth signal data, and can obtain the relationship between the Bluetooth signal data and the distance and time by using the data within Δt. The input X Δt of the recurrent neural network can be set as the Bluetooth signal receiving data within a period of time, that is:

[0066]

[0067] where Δt is the duration of receiving the Bluetooth signal data, n is the data size of receiving the Bluetooth signal data within Δt, is the ith input data of the Bluetooth signal data received within Δt. is a two-dimensional vector, where s is the size of the Bluetooth signal, and v is the time difference between receiving the i-1th data and receiving the ith data.

[0068] The output Y Δt of the recurrent neural network is a close contact probability vector, that is:

[0069]

[0070] Wherein, Δt is the duration of receiving Bluetooth signal data, m is the distance interval number, is the close contact probability under the distance interval i. This distance interval can be determined according to epidemic prevention requirements, for example: (0, 2) (2, 5) (5, 9) (9, +∞) (unit m), at this time m = 4, (0, 2) corresponds to distance interval 1, (2, 5) corresponds to distance interval 2, and so on.

[0071] When the close contact time inference model is trained, the edge user terminal can obtain the total contact time T Δt in Δt time through the received Bluetooth signal data. Δt The contact time in different distance intervals i Finally, the training label of the close contact time inference model can be obtained That is:

[0072]

[0073] When the close contact time is inferred through the close contact time inference model, for the output probability value of the recurrent neural network and the case where it is not 1, the output of the recurrent neural network is controlled using the Softmax function, so that the output probability value is 1. The mathematical operation of the Softmax function is as follows:

[0074]

[0075] is the close contact probability under the distance interval i obtained by the close contact time inference model. The edge user terminal can obtain the total contact time T Δt When the close contact probability of different distance intervals is output through the Softmax function , the total contact time T Δt can be obtained is multiplied by the total contact time T That is:

[0076]

[0077] The detection method utilizes the dynamic selectivity of reinforcement learning. In the federated learning environment, the weights of the edge user terminals are selected according to the local and global training process states of different edge user terminals, and a multi-agent reinforcement learning method V-value decomposition network (VDN) is used. Through the global reward obtained after each edge user terminal performs the weight selection action, the weight selection model of the edge user terminal can be iteratively trained.

[0078] The specific settings are as follows:

[0079] State: The state of each edge user terminal is composed of five parts: probe loss, global model accuracy, local data size, global data size, and local training rounds. Let represent the probe loss of the edge user terminal i in the tth training round, that is, the loss obtained by the edge user terminal using local data for training. The probe loss represents the difference between the data of the edge user terminal; the global model accuracy M represents the model accuracy effect after the edge user terminal selects the weight for parameter fusion, and also reflects the progress of training, and high accuracy indicates the later stage of training; the local data size k i and the global data size K potentially reflect the quantity relationship between the local and the global, providing a basis for weighting; the local training rounds P i refer to the number of local training times in one training process, and the more the training times, the more reason to believe that the weight should be increased; the state vector of the edge user terminal i in the tth training round is The setting of the state information reflects the training state of the local and the global.

[0080] Action: In each training process, the edge user terminal i selects the weight size of parameter fusion according to the state information. The action space is set as a t n ∈{1,2,3,4,5,6,7,8,9,10}, and the value in the range corresponds to a weight.

[0081] Reward: In order to reach the accuracy target as soon as possible, the reward signal of each iteration is set as the global model accuracy U t minus a set accuracy value u. The global model accuracy is the tight contact time inference model test accuracy of the edge user terminal i in the tth training, and n is the number of edge user terminals participating in the tth training, and maximizing the cumulative reward corresponds to finding the maximum model accuracy. The tth training reward r t =U t -u.

[0082] Specifically, the weight selection model of each edge user terminal uses a deep Q network (DQN) to infer its action. That is, a deep neural network (DNN) is used to realize the Q function Q θ n (s,a)=E[R t |s t n =s,a t n =a], where θ is the parameter of the DNN, is the total global discounted reward received by the edge user terminal n at the t-th training round. The edge user terminal n needs to use a replay buffer to save state transition tuples <s t n , a t n , s t+1 n , r t > for training locally using DQN. The reinforcement learning model training module updates the joint Q function . In this reinforcement learning environment, the state representation of the edge user terminal is global, so the Q network parameters are shared by all edge user terminals. The Q network of the edge user terminal can be trained on the central cloud server by minimizing the loss L = E st,at,rt,st+1 [y t -Q tot (s t ,a t )] 2 recursively, where is the parameter of the target network and is periodically copied during the training process.

[0083] After the reinforcement learning model training is completed, each edge user terminal can use the trained weight selection model to dynamically select the corresponding weight according to the current state of the environment to participate in the parameter fusion process to maximize the global test accuracy. At the same time, the dynamic selection of the reinforcement learning can reduce the weight of the edge user terminal with poor data quality, which can effectively suppress the data poisoning attack. In this method, the number of edge user terminals participating in the parameter fusion each time can be different, because the state of the edge user terminal contains global and local information, and each edge user terminal can select the optimal weight according to the state information.

[0084] Consider that there are n edge user terminals in the close contact detection system training process, let u i be the i-th edge user terminal, be the parameter of the close contact time inference model of the u i at the t-th training round, and the weight selected by the u i at the t-th training round be The global shared parameter after parameter fusion is θ t+1 , and the parameter fusion process can be represented as:

[0085]

[0086] The global shared parameter θ t+1 after parameter fusion will be distributed to the n edge user terminals for close contact time inference model accuracy testing.

[0087] The above describes exemplary embodiments of the present application, and does not limit the patent protection scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, based on the content of the present application specification and drawings, are also included in the patent protection scope of the present application.

Claims

1. A close contact detection method based on federated reinforcement learning, characterized in that, The close contact detection system based on federated reinforcement learning is divided into two parts: an edge user terminal and a central cloud server, two deep learning models are arranged in each edge user terminal, one is a close contact time inference model based on a recurrent neural network, and the other is a weight selection model in the parameter fusion process; the central cloud server includes three modules, namely a close contact network construction module, a parameter fusion module and a reinforcement learning model training module; the close contact time inference model has two parts, namely a recurrent neural network and a Softmax function, and the formula of the recurrent neural network is as follows: Wherein, Δt is the duration of receiving Bluetooth signal data, With respectively are the output of the i-1th and ith input data in the data received within Δt time, is the ith input data in the data received within Δt time, and θ is the parameter of the recurrent neural network; the edge user terminal stores the continuously received Bluetooth signal data, uses the data within Δt duration to obtain the relationship between the Bluetooth signal data and the distance and time, and the input X of the recurrent neural network is Δt The Bluetooth signal receiving data within a period of time is set, that is: wherein Δt is the duration of receiving the Bluetooth signal data, n is the data size of the Bluetooth signal data received in the Δt time, is the i-th input data of the Bluetooth signal data received in the Δt time, is a two-dimensional vector, wherein s is the size of the Bluetooth signal, and v is the time difference between receiving the i-1th data and receiving the i-th data; the output Y of the recurrent neural network Δt is a close probability vector, that is: Wherein, Δt is the duration period of receiving Bluetooth signal data, m is the distance interval number, is the close probability under the distance interval i; The close contact detection method includes close contact detection system training and close contact detection system use, and the close contact detection system training process specifically includes the following steps: Step 1, the edge user terminal generates a close contact identification pair after receiving the Bluetooth signal broadcast by other edge user terminals and immediately uploads it to the central cloud server to construct a close contact network; Step 2, the central cloud server selects the edge user terminal participating in this round of training; Step 3, the edge user terminal trains the close contact time inference model using local data, selects the weight according to the weight selection model, encrypts the parameters of the close contact time inference model using a homomorphic encryption algorithm after training, and uploads the encrypted parameters and weight information to the central cloud server; Step 4, after receiving the encrypted parameters and weight information uploaded by the edge user terminal, the central cloud server performs a parameter fusion process on the parameters, and distributes the fused global shared parameters to the edge user terminals participating in the training; Step 5, the edge user terminal updates the close contact time inference model using the fused global shared parameters, tests the updated close contact time inference model using the local test data set, and uploads the local test close contact time inference model accuracy to the central cloud server; Step 6, after receiving the close contact time inference model test accuracy of all edge user terminals participating in the training, the central cloud server forms a global reward, and the reinforcement learning model training module trains the weight selection model using the global reward, and distributes the trained weight selection model parameters to the edge user terminals participating in the training; The close contact detection system use process specifically includes the following steps: Step 1, the central cloud server determines the patient and the contact personnel according to the close contact network, and sends a close contact time inference instruction to the related edge user terminal; Step 2, the edge user terminal determines the user relationship according to the close contact time inference instruction, obtains the close contact time inference result of different distance intervals by using the close contact time inference model, and uploads the result to the central cloud server; Step 3, the central cloud server averages the uploaded close contact time inference results according to the corresponding relationship of the close contact identification pairs of the close contact network, and distributes the averaged results to the corresponding edge user terminals.

2. The federated reinforcement learning based close contact detection method according to claim 1, wherein, In the training process of the close contact detection system, when the close contact time inference model is trained, the edge user terminal obtains the total contact time T within the Δt time through the received Bluetooth signal data Δt The distance value between the edge user terminals is measured to obtain the total contact time T while the edge user terminal receives the Bluetooth signal of the other edge user terminal Δt The contact time under different distance intervals i Finally, the training label of the close contact time inference model can be obtained That is: When inferring the close contact time by the close contact time inference model, for the output probability value of the recurrent neural network and the case that is not 1, the Softmax function is used to control the output of the recurrent neural network, so that the output probability value is 1, and the mathematical operation of the Softmax function is as follows: i.e. the contact probability in distance interval i obtained by the contact time inference model, the edge user terminal obtains the total contact time T within Δt time through Bluetooth signal data Δt When outputting the contact probability of different distance intervals through the Softmax function Then, Multiply the total contact time T Δt The contact time in different distance intervals can be obtained That is:

3. The federated reinforcement learning based close contact detection method according to claim 2, wherein, The detection method utilizes dynamic selectivity of reinforcement learning, in a federated learning environment, for the local and global training process states of different edge user terminals, selects the weights of the edge user terminals, uses a multi-agent reinforcement learning method V value decomposition network, and trains the weight selection model of the edge user terminals through the global reward obtained after each edge user terminal performs the weight selection action.

4. The federated reinforcement learning based close contact detection method according to claim 3, wherein, The federated learning environment in the detection method is specifically set as follows: State: the state of each edge user terminal is composed of five parts: probe loss, global model accuracy, local data size, global data size, and local training rounds, set as represents the probe loss of edge user terminal i in the tth training round, that is, the loss obtained by the edge user terminal using local data for training. The probe loss represents the difference between the data of the edge user terminal. The global model accuracy M represents the model accuracy effect after the edge user terminal selects the weight for parameter fusion, and also reflects the progress of the training. High accuracy indicates that it is in the later stage of training. The local data size k i and the global data size K potentially reflect the quantity relationship between the local and the global, providing a basis for weighting. The local training rounds P i refer to the number of local training times in one training process. The more the training times, the greater the weight. The state vector of edge user terminal i in the tth training round is The setting of the state information reflects the training state of the local and the global. Action: In each training process, the edge user terminal i selects the weight size of parameter fusion according to the state information, and the action space is set as The value in the range corresponds to a weight value; Reward: To reach the precision target as quickly as possible, the reward signal for each iteration is set to the global model precision U t Subtract a set precision value u, the global model precision The edge user terminal i's close-in time inference model test precision for the tth round of training, n is the number of edge user terminals participating in the training for the tth round, maximizing the cumulative reward corresponds to finding the maximum model precision, the tth round of training reward r t = U t -u.

Images