A federated event sequence prediction method and system based on a time point process

Abstract

A kind of federal event sequence prediction method and system based on timing point process, its method includes: selected user and interaction scene (client), collect and process event data;Client initializes local partial model, including feature extraction model and event sequence model;User identifier is encrypted, and member alignment is carried out;The event feature of aligned local user is extracted, and event embedding vector is obtained;Sequential encryption event occurrence time is formed binary tuple with event embedding;According to time-event embedding binary tuple, restore the original order of user event embedding data;From original event embedding sequence, the extended vector required by each client training is extracted and returned;Client trains event sequence model, and the gradient required by parameter update of back propagation is updated;Local feature extraction model is updated, and gradient average is carried out to global event sequence model;From original event embedding, the extended vector corresponding to current time is returned, and the next occurrence event is predicted.

Description

TECHNICAL FIELD

[0001] The present application relates to a federated event sequence prediction method and system based on a time point process, belonging to the field of computer artificial intelligence. BACKGROUND

[0002] From e-commerce systems, financial transactions to medical diagnosis, the interaction of users and (virtual) scenes generates a large number of multi-event sequences. They are composed of a series of discrete events, each event is presented in the form of a two-tuple of occurrence time and event type. For example, a large number of patient diagnosis information in a hospital can be naturally regarded as different event sequences, each sequence records the disease diagnosis of a single patient over time.

[0003] With the rapid development of mobile network technology and wearable mobile devices, users can interact with various scenes at different time points, forming event sub-sequences distributed in different scenes. These scenes can be real-life places visited by users, or third-party applications provided by enterprises on mobile devices. Researchers can predict the future state of users by analyzing their historical behavior patterns. These predictions can have great value in real applications such as assisting medical diagnosis and business system traffic coordination. Event sequence modeling methods based on time point processes have become the mainstream technology in this research field.

[0004] However, due to the mutual influence between events in different scenes, the analysis of local data by institutions and enterprises will lead to a significant reduction in the accuracy of user behavior prediction. In addition, the user event data of different institutions is highly sensitive private data and cannot be centralized or shared.

[0005] Therefore, in order to solve the above problems, an efficient federated event sequence prediction scheme is needed, which can integrate scattered event information and accurately predict the future behavior of users without revealing privacy. SUMMARY

[0006] In view of the above problems and difficulties of the prior art, the present application provides a federated event sequence prediction method based on a time point process, which can integrate scattered event information and accurately predict the future behavior of users.

[0007] In a first aspect, the present application provides a federated event sequence prediction method based on a time point process, comprising the following steps:

[0008] S1: Selecting users and interaction scenes (clients), collecting and processing event data;

[0009] S2: The client initializes a local local model, including a feature extraction model and an event sequence model;

[0010] S3: encrypt user identity identifier and perform member alignment;

[0011] S4: extract event features of aligned local user to obtain event embedding vector;

[0012] S5: perform order-preserving encryption on event occurrence time and form binary tuple with event embedding;

[0013] S6: obtain event time-embedding binary tuple data scattered in each client and sort event embedding vectors according to encrypted event timestamp to restore original order of user event data;

[0014] S7: extract and return extended vector required for training of each client from original event embedding sequence according to design of event sequence model;

[0015] S8: the client continues to train the event sequence model using the extended vector and local event data, outputs probability of occurrence of local event subsequence, and back-propagates gradient required for parameter update;

[0016] S9: update local model, wherein the feature extraction model is directly updated by the optimizer, and the event sequence model is updated after gradient averaging operation;

[0017] S10: repeat S4-S9 until the joint model converges;

[0018] S11: in the prediction stage, perform S4-S6 to return extended vector corresponding to current time from original event embedding, and make prediction on next occurring event;

[0019] Preferably, the data described in S1 is a data set collected in a real scene, which can be collected by a smart watch, a mobile phone background record, or a preprocessed real public data set can be considered;

[0020] Preferably, the local model structure described in S2 is as follows:

[0021] The sequence feature extraction model based on Transformer includes an event embedding layer, an attention calculation layer, a normalization layer and a feedforward neural network layer, mainly uses a self-attention mechanism to encode the features of the event sequence, and compresses the original data while retaining the event information required for training;

[0022] The event sequence model based on time series point process includes a conditional intensity calculation layer, a log-likelihood calculation layer and a Monte Carlo sampling prediction layer;

[0023] Preferably, RSA asymmetric encryption algorithm is used in S3, and the user identity identifier needs to have uniqueness;

[0024] Preferably, the event embedding vector in S4 is obtained by sequence feature extraction from the local event data of each client, and each local event has a corresponding event embedding;

[0025] Preferably, an OPE order-preserving encryption algorithm is used in S5 to ensure that the original time order of the event embedding can be restored and specific user time-sensitive information is not disclosed;

[0026] Preferably, a new extended (hidden) vector calculation method is used in S7, which extracts dispersed event information and captures low-order information of the sequence by aggregating the nearest embedding and neighbor events of the predicted event on each client;

[0027] Preferably, the specific process of S8 is as follows:

[0028] The event sequence modeling adopts a probabilistic model based on a timing point process, and the conditional intensity function is selected from a corrected random process form, and the (conditional) log probability likelihood of the occurrence of the event sequence is trained;

[0029] The training process adopts a static embedding update strategy to realize the back propagation of the gradient, that is, only the local embedding is updated in the training process, and the embedding gradient of other clients is truncated to 0;

[0030] Preferably, the parameters of the event sequence model are updated in S9 using the traditional federated averaging method to realize the global sharing of the client event sequence model, while the feature extraction model is local and directly updated;

[0031] Preferably, the specific process of S11 is as follows:

[0032] After obtaining the original event embedding sequence by repeating steps S4-S6, only the last event embedding of each client and the neighbor embedding of the event to be predicted are extracted to obtain an extended vector, which is returned to each client;

[0033] Since the event sequence model is globally shared, the same prediction can be obtained by each client according to the extended vector, and the Monte Carlo algorithm and the maximum likelihood estimation are mainly used to predict the time and type of the future event occurrence, respectively.

[0034] In a second aspect, the application also provides a system for implementing a federated event sequence prediction method based on a timing point process, the system comprising local clients deployed in each scene, a matcher for member alignment, a third-party encryption server for encrypting private information, a third-party reconstruction server for restoring the order of event embedding, and a global server for federated averaging;

[0035] The client is responsible for deploying the aforementioned feature extraction model and event sequence model, responsible for encrypting the user identity and passing it to the matcher, calculating the event embedding through the aligned event data of the user, encrypting the event timestamp in order and passing it to the reconstruction server, and receiving the expansion vector and training the event sequence model;

[0036] The matcher is responsible for receiving all the encrypted identity identifiers of the users for member alignment, and returning the aligned user identity to the client in the training stage.

[0037] The encryption server is responsible for generating the public key of the encryption algorithm and passing it to the client, and the client encrypts the timestamp and user identifier according to the public key;

[0038] The reconstruction server includes a comparator, a memory, a selector and an accumulator, wherein the memory and the comparator are used to store the event embedding vector and the sorted encrypted timestamp respectively, and the selector and the accumulator generate the event embedding according to the generation method of the expansion vector;

[0039] The global server receives the parameter gradient of the event sequence model of each client, and returns it to the client after gradient averaging for updating.

[0040] As can be seen from the above, when the scheme of the application is applied to event sequence joint learning in multiple scenarios, first, the local model of each party encodes the event subsequence into an embedding vector. Then, the embedding sequence reconstructor restores the order of these embeddings by sorting the encrypted timestamps from different parties, and returns the expansion vector for further training. Finally, through a specially designed global event sequence model, each party can calculate the log-likelihood probability and jointly predict the next event.

[0041] The advantages of the application can be summarized as follows: by solving the challenge of time correlation of multiple scenario events, a new federated learning framework for event sequence prediction is proposed, which significantly improves the accuracy of user behavior prediction, and strictly protects user privacy data, avoiding the risk of privacy leakage. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0043] Figure 1 is a flowchart of a federated event sequence prediction method based on time series point process provided by an embodiment of the present application.

[0044] Figure 2 is a structural schematic diagram of a federal event sequence prediction system based on a timing point process provided by an embodiment of the present application. DETAILED DESCRIPTION

[0045] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. The technical solutions in the embodiments of the present application will be described clearly and completely below. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application belong to the scope of protection of the present application.

[0046] Referring to Figure 1 , Figure 1 A user disease prediction method based on a federal event sequence prediction method based on a timing point process provided by an embodiment of the present application is provided. The above method comprises the following steps S101-S107:

[0047] Step S101: Collect and process event data.

[0048] Realistic application scenarios of event prediction are widespread. Taking user disease prediction as an example, the task of federated learning is to infer the user's possible medical behavior and potential disease type according to the user's independent medical data in different hospitals or different departments. This embodiment uses a public data set collected in a real scenario, Mimic medical data set, which is developed based on an electronic medical record system and contains seven years of clinical medical records of anonymous patients. Each patient represents a separate sequence, and the events in the sequence are composed of medical diagnosis and medical diagnosis.

[0049] This embodiment randomly divides events into different scenarios according to event categories to simulate the distribution of real multi-scenario event sequence data.

[0050] Step S102: Align members.

[0051] This embodiment uses the sequence number in the real data set as the user identifier, and the event sub-sequences obtained by random splitting belong to the same user and share the same identifier. Each client encrypts the user identifier using the RSA asymmetric encryption algorithm. Then, the ciphertext is compared by a third-party matcher to filter out user sequence samples that can be trained together. Finally, the client performs federated learning based on the aligned samples.

[0052] Step S103: Client encodes event sequence.

[0053] The sequence feature extraction module based on the Transformer is composed of an event embedding layer, an attention calculation layer, a normalization layer, and a feedforward neural network layer, as follows:

[0054] First, the position embedding is calculated as follows according to the time when the local event data occurs:

[0055]

[0056] Then, the type embedding is retrieved according to the embedding table The final event is encoded as the sum of the position embedding and the type embedding where D denotes the dimension of the model, i denotes the aligned customer number, n denotes the event sequence number, and k denotes the client number;

[0057] For the encoded event data First, it is mapped to Then, it is input into a feed-forward neural network layer and layer normalized, as follows:

[0058]

[0059] H i,k = Feed-forward(H i,k ), (4)

[0060] where Q, denotes the query, key, and value in the self-attention calculation process, denotes the parameter matrix of linear projection in the self-attention mechanism.

[0061] Step S104: Embedding sequence reconstruction.

[0062] Through the local feature extraction model, each event generates a corresponding event embedding. The OPE order-preserving encryption technology can encrypt the time when the event embedding occurs, so as to facilitate the reconstruction server to restore the original order of each user's event embedding. This can be specifically represented as:

[0063]

[0064] where denotes a rearrangement operation, and the event embeddings H i,1 ,…,H i,K of user i on different clients are rearranged according to their timestamps to obtain the final embedding matrix H i .

[0065] Step S105: Extension (hidden) vector generation.

[0066] In order to reduce the communication volume of back propagation, the present embodiment adopts a novel extension vector calculation method, which can be represented in the form of neighbor item and aggregation item accumulation: ​

[0067]

[0068] where C is the number of neighbors, l(t) denotes the predecessor index of the current sequence closest to time t, H i is the embedding matrix obtained by the rearrangement operation, is the extended embedding (hidden representation) matrix required by client k. Local embedding is dynamically updated by client k, while the remaining embeddings remain static.

[0069] Step S106: Model training.

[0070] Event modeling takes a probabilistic model based on a time-series point process, whose objective function is the product of the conditional probabilities of event occurrence, which can be expressed as:

[0071]

[0072] where the log-likelihood of the event log Pr(S i ) is the sum of the local log-likelihoods of K clients. Each client can calculate the probability of local event occurrence according to its extended vector:

[0073]

[0074] where β, μ are scalars, and are parameter matrices. Finally, the gradients of the model parameters are back-propagated according to the calculation results, and the model is trained and updated. There is no need for information exchange between clients in this stage, and the local model parameters of each client can be updated.

[0075] Train the federated model, repeat steps S103-S106 until the preset number of iterations is reached, or the stopping condition set to prevent overfitting is reached.

[0076] Step S107: Joint prediction of user diseases.

[0077] When a new user has a medical need, input the user's historical events into the trained joint model to obtain the disease prediction result of the user, and then provide auxiliary medical advice for the user. This embodiment specifically verifies the effectiveness of the federated model on the test user set. Given the historical event sequence S i of user i, the time and type of the N i +1th (predicted) event occurrence can be expressed as:

[0078]

[0079] where Pr(Δt|S i), Pr(e|S i ) is consistent with step S106. The generation of the expansion vector only needs to use the previous index of the current time .

[0080] Corresponding to the above-mentioned federal event sequence prediction method based on the timing point process, the embodiment of the application also provides a federal event sequence prediction system based on the timing point process.

[0081] In an embodiment of the application, referring to Figure 2 , a structural schematic diagram of a federal event sequence prediction system based on the timing point process is provided, and the system comprises an encryption server 201, a member alignment matcher 202, a local model 203, a reconstructor 204 and a gradient aggregation server 205.

[0082] The encryption server 201 is composed of an RSA encryption port and an OPE encryption port; each encryption port is responsible for distributing the public key of the corresponding encryption algorithm and only communicates with the client in a simple manner.

[0083] The member alignment matcher 202 is composed of a receiver and a comparator; the receiver receives the encrypted user identity sent by the client, and the comparator is responsible for matching and aligning the same user information and returning the aligned user to the client.

[0084] The local model 203 is composed of a feature extraction model, user event sequence data and an event sequence model; the feature extraction model is responsible for encoding the event embedding vector, and the event sequence model is responsible for the probability reconstruction of the event sequence according to the expansion vector.

[0085] The reconstructor 204 is composed of a memory, a comparator, a selector and an accumulator; the memory receives the encrypted event embedding vector, the comparator restores the original timing sequence of the event embedding, and the selector and the accumulator realize the generation operation of the expansion vector.

[0086] The gradient aggregation server 205 receives the event sequence model parameter gradient uploaded by each client, returns each end after gradient averaging, and realizes the purpose of global sharing of the event sequence model.

[0087] As can be seen from the above, the application realizes a federal event sequence prediction method and system based on the timing point process. By integrating the dispersed event information of each party, the application can effectively improve the accuracy and security of event prediction.

[0088] The above only describes the preferred embodiments of the application and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A federated event sequence prediction method based on a timing point process, comprising the following steps: S1: Selecting a user and an interaction scenario, collecting and processing event data; S2: Client initializes a local local model, including a feature extraction model and an event sequence model; S3: Encrypt the user identifier and perform member alignment; S4: Extract the event features of the aligned local user to obtain the event embedding vector; S5: Order-preserving encryption event occurrence time, and form a binary tuple with the event embedding; S6: Obtain the event time-embedding binary tuple data distributed on each client, and sort the event embedding vectors according to the encrypted event timestamp to restore the original order of the user event data; S7: According to the design of the event sequence model, extract and return the extended vector required for each client training from the original event embedding sequence; S8: The client uses the extended vector and local event data to continue training the event sequence model, outputs the probability of the occurrence of the local event subsequence, and back propagates the gradient required for parameter updating; S9: Update the local model, wherein the feature extraction model is directly updated by the optimizer, and the event sequence model also needs to be updated after gradient averaging operation; S10: Repeat S4-S9 until the joint model converges; S11: In the prediction stage, execute S4-S6 to return the extended vector corresponding to the current time from the original event embedding, and make a prediction for the next occurring event.

2. The federated event sequence prediction method based on time point process according to claim 1, characterized in that, The event data in step S1 is a data set collected in a real scene, which is collected by an intelligent watch, a mobile phone background record, or a preprocessed real public data set.

3. The federated event sequence prediction method based on time-point process according to claim 1, wherein, The local model structure in step S2 is specifically: A sequence feature extraction model based on Transformer, which includes an event embedding layer, an attention calculation layer, a normalization layer and a feedforward neural network layer, which encodes the features of the event sequence using a self-attention mechanism, compresses the original data while retaining the event information required for training; An event sequence model based on a timing point process, which includes a conditional intensity calculation layer, a log likelihood calculation layer and a Monte Carlo sampling prediction layer.

4. The federated event sequence prediction method based on time-point process according to claim 1, wherein, In step S3, RSA asymmetric encryption algorithm is used, and the user identifier needs to be unique; In step S4, the event embedding vector is obtained by sequence feature extraction from the local event data of each client, and each local event has a corresponding event embedding; In step S5, the OPE order-preserving encryption algorithm is used to ensure that the original time order of the event embedding can be restored, and the specific user time sensitive information is not leaked.

5. The federated event sequence prediction method based on time-point process according to claim 1, wherein, In step S7, an extended vector calculation method is used to aggregate the nearest embedding and neighbor event embedding of the predicted event on each client to extract the dispersed event information and capture the low-order information of the sequence.

6. The federated event sequence prediction method based on time-point process according to claim 1, wherein, The specific process of training in step S8 is: The event sequence modeling adopts a probability model based on a timing point process, the conditional intensity function is selected from the form of a corrected random process, and the log probability likelihood of the event sequence occurrence is trained; The training process adopts a static embedding update strategy to realize the back propagation of the gradient, that is, only the local embedding is updated during the training process, and the event embedding gradient of other clients is truncated to 0.

7. The federated event sequence prediction method based on time-point process according to claim 1, wherein, The parameters of the event sequence model are updated in step S9 using the traditional federated averaging method to achieve global sharing of the client event sequence model, while the feature extraction model is local and directly updated.

8. The federated event sequence prediction method based on time-point process according to claim 1, wherein, The specific process of prediction in step S11 is as follows: After repeating steps S4-S6 to obtain the original event embedding sequence, only the last event embedding of each client and the neighbor embedding of the event to be predicted are extracted to obtain an extended vector, which is returned to each client. Since the event sequence model is globally shared, the clients can obtain the same prediction based on the extended vector. The Monte Carlo algorithm and maximum likelihood estimation are mainly used to predict the time and type of future events.

9. A system for implementing the federated event sequence prediction method based on the time series point process according to claim 1, wherein the system comprises local clients deployed in each scene, a matcher for member alignment, a third-party encryption server for encrypting private information, a third-party reconstruction server for restoring the order of event embeddings, and a global server for federated averaging; The client is responsible for deploying the aforementioned feature extraction model and event sequence model, encrypts the user identity and passes it to the matcher, calculates the event embedding based on the event data of the aligned user, encrypts the event timestamp in order and passes it to the reconstruction server, and receives the extended vector and trains the event sequence model; The matcher is responsible for receiving the encrypted identity of all users for member alignment, and returning the aligned user identity to the client during the training phase; The encryption server is responsible for generating the public key of the encryption algorithm and passing it to the client, and the client encrypts the timestamp and user identity identifier based on the public key; The reconstruction server includes a comparator, a memory, a selector, and an accumulator, wherein, The memory and comparator are used to store the event embedding vector and the sorted encrypted timestamp, respectively, and the selector and accumulator generate the event embedding based on the generation method of the extended vector; The global server receives the parameter gradient of the event sequence model of each client, performs gradient averaging, and returns it to the client for updating.

Images