User behavior prediction method and device, electronic equipment and medium

Abstract

The embodiment of the invention relates to the technical field of big data, and provides a user behavior prediction method and device, electronic equipment and a medium, and the method comprises the steps: obtaining first historical behavior data and a target prediction task of a target user; inputting the first historical behavior data into a pre-trained user behavior prediction model based on the target prediction task to predict a future target behavior; wherein the user behavior prediction model obtains second historical behavior data and third historical behavior data of a sample user, and the third historical behavior data is future behavior data corresponding to the second historical behavior data; extracting historical behavior characteristics of the sample user based on the second historical behavior data; and inputting the historical behavior characteristics and the third historical behavior data into the initial model for training, optimizing hyper-parameters of the initial model in the training process, and obtaining a user behavior prediction model until the accuracy of an output result of the initial model meets a condition. Therefore, accurate prediction of user behaviors is realized.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a user behavior prediction method, device, electronic device and medium. Background Art

[0002] With the development of machine learning technology, it has been widely applied to solve problems in various technical fields. User behavior prediction, as an application of machine learning, can predict future user behavior based on collected user behavior data, thereby making appropriate decisions, such as pushing relevant services. Therefore, how to accurately predict user behavior has become a pressing issue.

[0003] Currently, machine learning algorithms are primarily used to predict user behavior as a sequential recommendation problem, leveraging historical user behavior sequences to predict future user behavior. A commonly used traditional approach is the Markov process, which assumes that current behavior is only related to previous behavior, simplifying the complex sequence modeling process. First, all possible user behaviors are defined as states. Then, based on historical data, transition probabilities between different states are calculated to construct a state transition matrix. This matrix can be used to predict a user's next behavior or generate a sequence of behaviors, making it suitable for short-term behavior prediction and recommendation systems. However, these methods only consider short-term dependencies in user behavior and struggle to capture complex relationships in longer sequences. As the number of user behavior types increases, the state space rapidly expands, making it difficult to accurately estimate transition probabilities. Furthermore, Markov processes struggle to incorporate multiple factors, such as time and context, and require a high data volume. Using models such as graph neural networks (GNNs) and regressive neural networks (RNNs), GNNs can represent users and their behaviors as graph structures, capturing relationships and multi-hop connections between different behaviors, thereby revealing more complex user behavior dependencies. RNNs process user behavior sequences step by step through a serialized approach, leveraging their memory capacity to capture long-term temporal dependencies. While GNNs excel at handling complex relationships within graph structures, they struggle to capture temporal dependencies and have high computational complexity, making them difficult to handle large-scale data. RNNs can handle temporal dependencies, but are prone to vanishing or exploding gradients in long sequences, making them incapable of memorizing further historical information. Furthermore, RNNs are sensitive to noise in sequences, leading to unstable model performance.

[0004] Existing methods face significant challenges in processing large-scale user behavior data, especially when faced with complex and diverse behavior patterns. Although GNNs and RNNs can capture sequential dependencies or relationships in graph structures, they have limitations in scalability. It is difficult to effectively increase the parameter size to capture the more detailed characteristics of behavior sequences by simply stacking models. In addition, as model complexity increases, training time, memory usage, and computational overhead also increase significantly, making efficient modeling difficult in large-scale data environments. To effectively characterize the complexity and diversity of user behavior, it is necessary to explore more scalable and efficient model architectures or innovative optimization methods.

[0005] Existing technologies face the following limitations when modeling user behavior sequences: (1) Existing patents often ignore the problem of uneven distribution of user behaviors, especially when dealing with long-tail user intentions. These long-tail intentions refer to rare or uncommon behaviors of a small number of users. Due to data sparsity, modeling and predicting these behaviors becomes extremely difficult. The lack of data limits the model's ability to learn uncommon user intentions, making it difficult to accurately capture the complex patterns therein, resulting in poor performance when generalizing to other areas of user behavior; (2) The model is sensitive to noise in the behavior sequence and is easily disturbed by accidental behaviors or data anomalies, resulting in unstable model performance and affecting its application effect in real scenarios; (3) Existing patents often focus on a certain area of ​​user behavior, and user behavior patterns are relatively single. They are often unable to fit the complex and diverse behavior patterns of users at all times and all day long, making it difficult to flexibly characterize users' multiple behavioral preferences and situational changes, thereby affecting the accurate prediction of user behavior; (4) Existing models lack scalability, especially when facing large-scale user behavior data. Model capabilities cannot be improved by increasing model parameters or simply stacking, which limits their practicality in big data scenarios. Summary of the Invention

[0006] The present invention provides a user behavior prediction method, device, electronic device and medium to address the defects of the existing technology such as inaccurate user behavior prediction and unstable performance that affect application effects, achieve accurate and fair user behavior prediction effects, and improve the generalization ability of the model.

[0007] The present invention provides a user behavior prediction method, comprising: Obtain the target user's first historical behavior data and target prediction tasks; Inputting the first historical behavior data into a pre-trained user behavior prediction model based on the target prediction task, and predicting the target user's future target behavior through the user behavior prediction model, wherein the future target behavior includes historical behaviors, historical behaviors that have not occurred, and long-term behaviors; The user behavior prediction model is trained based on the following steps: Obtaining second historical behavior data and third historical behavior data of the sample user, wherein the historical behavior data includes date information, time period information, location information, and event information, and the third historical behavior data is future behavior data corresponding to the second historical behavior data; extracting the historical behavior features of the sample user based on the date information, time period information, location information, and event information in the second historical behavior data; The historical behavior characteristics and the third historical behavior data are input into the initial model for training. During the training process, the hyperparameters of the initial model are optimized until the accuracy of the output result of the initial model meets the conditions. The model training is determined to be completed and a user behavior prediction model is obtained.

[0008] In one possible implementation, the user behavior prediction model includes a day embedding layer, a timestamp embedding layer, a location embedding layer, and an event embedding layer, and the historical behavior features include date features, time features, location features, and event features; The date information is converted into a corresponding date feature embedding vector with the corresponding week ID through the day embedding layer; The time period information is divided into multiple time periods according to the time stamp embedding layer, and the time period ID corresponding to each time period is converted into a corresponding time feature embedding vector; Converting the geographic location ID corresponding to the location information into a corresponding location feature embedding vector through the location embedding layer; The event features of the event information are extracted through the event embedding layer, and the event features are converted into event feature embedding vectors.

[0009] In one possible implementation, the method further includes: Extracting the dependency relationship among the date feature embedding vector, the time feature embedding vector, the location feature embedding vector, and the event feature embedding vector through an attention mechanism, and obtaining a mapping function of the historical event sequence corresponding to the second historical behavior data; Training the initial model based on the mapping function and the third historical behavior data; The hyperparameters of the initial model are set, and during the model training process, the hyperparameters are optimized based on the results of each round of training.

[0010] In one possible implementation, the user behavior prediction model further includes a prediction layer; Using a multilayer perceptron to learn the mapping function to obtain an output result; The output result is compared with the third historical behavior data. When the accuracy of the output result is greater than a threshold, it is determined that the initial model training is completed, and the trained initial model is used as a user behavior prediction model.

[0011] In one possible implementation, the method further includes: Based on the preset model training mechanism and the results of each round of training, the hyperparameters are optimized using the distributed robust optimization method.

[0012] In one possible implementation, the method further includes: If the target prediction task is a future behavior prediction task, the first historical behavior data is input into a pre-trained user behavior prediction model, and the future behavior of the target user is predicted by the user behavior prediction model, where the future behavior is a historical behavior; If the target prediction task is a new behavior prediction task, the user behavior prediction model is trained again based on behavior data that has not occurred in the past; Inputting the first historical behavior data into the twice-trained user behavior prediction model, and predicting the target user's future new behavior using the twice-trained user behavior prediction model; If the target prediction task is a long-term behavior generation task, the first historical behavior data is input into a pre-trained user behavior prediction model, and future behaviors are generated in an autoregressive manner through the user behavior prediction model and extended to a preset sequence length.

[0013] The present invention also provides a user behavior prediction device, comprising the following modules: An acquisition module, used to obtain the first historical behavior data and target prediction task of the target user; A prediction module, configured to input the first historical behavior data into a pre-trained user behavior prediction model based on the target prediction task, and predict the target user's future target behavior using the user behavior prediction model, wherein the future target behavior includes historical behaviors, historical behaviors that have not occurred, and long-term behaviors; A model training module is used to obtain the second historical behavior data and third historical behavior data of the sample user, wherein the historical behavior data includes date information, time period information, location information and event information, and the third historical behavior data is the future behavior data corresponding to the second historical behavior data; based on the date information, time period information, location information and event information in the second historical behavior data, the historical behavior characteristics of the sample user are extracted; the historical behavior characteristics and the third historical behavior data are input into the initial model for training, and the hyperparameters of the initial model are optimized during the training process until the accuracy of the output result of the initial model meets the conditions, the model training is determined to be completed, and a user behavior prediction model is obtained.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the user behavior prediction method as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the user behavior prediction methods described above.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the user behavior prediction methods described above.

[0017] The user behavior prediction method, device, electronic device and medium provided by the present invention obtain the first historical behavior data and target prediction task of the target user; input the first historical behavior data into a pre-trained user behavior prediction model based on the target prediction task, and predict the future target behavior of the target user through the user behavior prediction model, wherein the future target behavior includes behaviors that have occurred in the past, behaviors that have not occurred in the past and long-term behaviors; wherein the user behavior prediction model is trained based on the following steps: obtaining the second historical behavior data and third historical behavior data of the sample user, wherein the historical behavior data includes date information, time period information, location information and event information, and the third historical behavior data is the future behavior data corresponding to the second historical behavior data; extracting the historical behavior characteristics of the sample user based on the date information, time period information, location information and event information in the second historical behavior data; inputting the historical behavior characteristics and the third historical behavior data into the initial model for training, and optimizing the hyperparameters of the initial model during the training process until the accuracy of the output result of the initial model meets the conditions, determining that the model training is completed, and obtaining the user behavior prediction model. Compared with the defects of inaccurate user behavior prediction and unstable performance in existing technologies that affect application effects, this solution can achieve accurate and fair user behavior prediction and improve the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a flow chart of the user behavior prediction method provided by the present invention.

[0020] Figure 2 It is a flowchart of the training method of the user behavior prediction model provided by the present invention.

[0021] Figure 3 It is a structural diagram of the user behavior prediction model provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the cross-domain adaptation of the user behavior prediction model provided by the present invention.

[0023] Figure 5 This is the flowchart of the basic model pre-training and downstream task migration provided by the present invention.

[0024] Figure 6This is a long-term behavior generation flow chart provided by the present invention.

[0025] Figure 7 It is a structural diagram of the user behavior prediction device provided by the present invention.

[0026] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0028] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below with reference to the accompanying drawings. The embodiments do not limit the embodiments of the present invention.

[0029] Figure 1 is a flow chart of the user behavior prediction method provided by the present invention, such as Figure 1 As shown, the method includes the following: S11. Obtain the first historical behavior data and target prediction task of the target user.

[0030] The goal of the embodiments of the present invention is to accurately predict the future behavior of users. Accurate user behavior prediction can greatly improve personalized services and user experience, thereby enhancing user engagement and loyalty, while optimizing resource allocation and improving system operational efficiency. In addition, it can provide companies with important user insights, support more informed business decisions, and improve marketing effectiveness and conversion rates through precise advertising, thereby bringing higher returns on investment and competitive advantages. This method is applied to a basic model for large-scale user behavior modeling, BehaviorGPT (the user behavior prediction model of the embodiment of the present invention), and through a novel pre-training paradigm, it makes user behavior prediction more fair, improves the effect of long-tail behavior, and enhances the generalization ability of the model.

[0031] Specifically, a user behavior prediction model needs to be pre-trained. The model input is the historical behavior data of the target user, which includes features such as date information, time period information, location information, and event information. Each feature is processed through the corresponding embedding layer to obtain an embedding vector. Then, these embedding vectors are integrated and the relationship between different features is captured through the attention mechanism (Transformer) block. Finally, the prediction layer uses this learned information to predict the user's future behavior. The training process and model structure of this user behavior prediction model are described in Figure 2 The corresponding embodiments are described in detail and will not be described in detail here.

[0032] S12. Based on the target prediction task, the first historical behavior data is input into a pre-trained user behavior prediction model, and the future target behavior of the target user is predicted by the user behavior prediction model.

[0033] Target prediction tasks include future behavior prediction tasks, new behavior prediction tasks, and long-term behavior generation tasks. The pre-trained user behavior prediction model needs to be set according to different target prediction tasks.

[0034] For example, if the target prediction task is a future behavior prediction task, the first historical behavior data is input into a pre-trained user behavior prediction model, and the user behavior prediction model is used to predict the future behavior of the target user, where the future behavior is the behavior that has occurred in the past.

[0035] If the target prediction task is a new behavior prediction task, the user behavior prediction model is trained again based on behavior data that has not occurred in the past.

[0036] The first historical behavior data is input into the user behavior prediction model after secondary training, and the future new behavior of the target user is predicted by the user behavior prediction model after secondary training.

[0037] If the target prediction task is a long-term behavior generation task, the first historical behavior data is input into the pre-trained user behavior prediction model, and the future behavior is generated in an autoregressive manner through the user behavior prediction model and extended to a preset sequence length.

[0038] The user behavior prediction method provided by the present invention obtains the first historical behavior data and target prediction task of the target user; inputs the first historical behavior data into a pre-trained user behavior prediction model based on the target prediction task, and predicts the future target behavior of the target user through the user behavior prediction model, wherein the future target behavior includes behaviors that have occurred in the past, behaviors that have not occurred in the past, and long-term behaviors; wherein the user behavior prediction model is trained based on the following steps: obtaining the second historical behavior data and the third historical behavior data of the sample user, wherein the historical behavior data includes date information, time period information, location information and event information, and the third historical behavior data is the future behavior data corresponding to the second historical behavior data; extracting the historical behavior features of the sample user based on the date information, time period information, location information and event information in the second historical behavior data; inputting the historical behavior features and the third historical behavior data into the initial model for training, and optimizing the hyperparameters of the initial model during the training process until the accuracy of the output result of the initial model meets the conditions, determining that the model training is completed, and obtaining the user behavior prediction model. Compared with the defects of inaccurate user behavior prediction and unstable performance in existing technologies that affect application effects, this method can achieve accurate and fair user behavior prediction and improve the generalization ability of the model.

[0039] Figure 2 This is a flow chart of the training method of the user behavior prediction model provided by the present invention. Figure 2 As shown, the method includes the following: This embodiment of the present invention designs a basic model for large-scale user behavior modeling. The model inputs include features such as day of week, timestamp, location, and historical events. Each feature is processed through a corresponding embedding layer. These embeddings are then combined and used in a Transformer block to capture the relationships between different features. Finally, the prediction layer uses this learned information to predict future user behavior.

[0040] Furthermore, the problem of predicting user future behavior can be expressed as follows: User behavior can be expressed as ,in Represents a user On date , time is and location A specific event that occurred on the They correspond to the week ID, time period ID, location ID and event ID respectively. Represents the collection of week, time period, location and event, and the corresponding data size is , used to verify the legality of the data input to the embedding layer. The goal of user behavior prediction is to An event sequence is used to predict future user behavior, which can be expressed as: What the model needs to learn is the mapping function During model training, you also need to set model hyperparameters (e.g., feature vector dimensions, learning rate, and hidden layer dimensions of the attention aggregation network). During model training, the weights and biases of each layer of the network can be updated using the Adam optimization algorithm during backpropagation.

[0041] S21. Obtain the second historical behavior data and the third historical behavior data of the sample user.

[0042] This embodiment of the present invention describes in detail the training method for the user behavior prediction model used in this user behavior prediction method. First, sample data is obtained for model training. The sample data consists of the second and third historical behavior data of the sample user. The historical behavior data includes date information, time period information, location information, and event information. The third historical behavior data is the future behavior data corresponding to the second historical behavior data.

[0043] S22. Extract historical behavior features of the sample user based on date information, time period information, location information, and event information in the second historical behavior data.

[0044] like Figure 3 As shown, the user behavior prediction model includes a day embedding layer, a timestamp embedding layer, a location embedding layer and an event embedding layer, and the historical behavior features include date features, time features, location features and event features.

[0045] The day embedding layer converts the date information into the corresponding date feature embedding vector with the corresponding week ID; the timestamp embedding layer converts the time period information into the corresponding time period ID of multiple time periods divided by 24 hours a day into the corresponding time feature embedding vector; the location embedding layer converts the geographic location ID corresponding to the location information into the corresponding location feature embedding vector; the event embedding layer extracts the event features of the event information and converts the event features into the event feature embedding vector.

[0046] Specifically, the user behavior prediction model is designed with four embedding layers to capture different features: day embedding layer, timestamp embedding layer, location embedding layer, and event embedding layer. Each embedding layer can convert discrete ID features into an embedding vector.

[0047] The daily embedding layer converts the discrete ID representations of the seven days of the week (Monday through Sunday) into corresponding embedding vectors. This embedding helps the model identify and distinguish user behavior patterns throughout the week, thereby capturing possible cyclical patterns. For example, differences in user behavior between weekdays and weekends, or activity trends on specific days, can be observed.

[0048] The timestamp embedding layer converts the discrete ID features of 96 15-minute intervals throughout the day into corresponding embedding vectors, expressing time period information in a fine-grained manner. This representation helps the model identify short-term behavioral patterns within a day, such as user activity during specific time periods and behavioral differences during peak hours in the morning and evening.

[0049] The location embedding layer is used to represent different geographic location information. By converting discrete geographic location ID features into embedding vectors, this layer helps capture spatial associations and patterns, enabling the model to understand possible connections or differences between different locations.

[0050] The event embedding layer is specifically used to represent user behavior or event information. By converting these event features into embedding vectors, the event embedding layer can help the model identify user behavior patterns and potential relationships between different events.

[0051] The user behavior prediction model also includes a Transformer block: This block consists of N identical layers stacked together. Each layer uses an attention mechanism to capture increasingly complex feature interactions. This layer-by-layer structure allows the model to gradually learn high-order dependencies between input features, thereby forming an implicit representation of historical event sequences.

[0052] in, are the embedding vectors corresponding to the output of the above four embedding layers.

[0053] However, the existing common Transformer architecture has a computational and memory complexity of ,in is the sequence length. As sequence length increases, complexity increases quadratically, making it difficult for the Transformer to efficiently process long sequences. This is especially true when stacking multiple layers, where the computational burden is further increased. To address this issue, this model introduces Flash Attention, a new, faster and memory-efficient attention mechanism. This allows the model to more effectively process long sequences while reducing computational overhead, thereby achieving more efficient performance in long sequence modeling.

[0054] S23. Input the historical behavior characteristics and the third historical behavior data into the initial model for training. During the training process, optimize the hyperparameters of the initial model until the accuracy of the output result of the initial model meets the conditions. Then, determine that the model training is completed and obtain the user behavior prediction model.

[0055] The user behavior prediction model also includes a prediction layer, which uses a multi-layer perceptron to learn the mapping function and obtain an output result; the output result is compared with the third historical behavior data. When the accuracy of the output result is greater than a threshold (for example, 99%), it is determined that the initial model training is completed, and the trained initial model is used as the user behavior prediction model.

[0056] Specifically, a multi-layer perceptron (MLP) is used as the prediction layer, and its structure can be expressed as: in, are learnable parameters.

[0057] The embodiment of the present invention also proposes a model pre-training paradigm designed specifically for user behavior data, which aims to achieve fair modeling of user behavior and significantly improve the generalization ability of the model. This paradigm minimizes the impact of the uncertainty of the tail category distribution on the model, enabling the model to perform well on both head and tail behaviors, thereby achieving more balanced and robust predictions. Head category behaviors generally refer to user behaviors that occur frequently, are influential, or are of high value. These behaviors are often concentrated in a small number of core user groups and have a significant impact on the overall performance of the platform or product.

[0058] For example: high-frequency usage behaviors: such as frequent logins, long stays, high-frequency purchases, etc.

[0059] High-value behaviors: such as large-scale consumption, use of key functions, and creation of high-quality content.

[0060] High-impact behaviors: such as posting popular content on social platforms and participating in important discussions.

[0061] Tail category behaviors refer to user behaviors that occur less frequently, have less influence, or have lower value. These behaviors are usually distributed among a large number of users, but the contribution of each user's behavior is relatively small.

[0062] For example: low-frequency usage behaviors: such as occasional login, short stay, and very few purchases.

[0063] Low-value behaviors: such as small amounts of interaction and low-quality content consumption.

[0064] Marginal behavior: such as users' use of non-core functions on the platform, occasional browsing, etc.

[0065] To this end, we introduce distributionally robust optimization (DRO), an optimization method specifically designed to handle tasks with inherent uncertainty. DRO allows for differences in the distribution of training and test data within a predefined uncertainty set, thereby enhancing the model's adaptability and robustness to distribution shifts. The optimization formula can be expressed as: in, ℓ Denotes the cross entropy loss. For head category behaviors, their data is relatively abundant, and the model can estimate their scaling factors with high confidence, resulting in a smaller uncertainty set. In contrast, for tail category behaviors, due to the scarcity of data, the model has difficulty in accurately estimating, resulting in a larger uncertainty set.

[0066] This pre-training mechanism focuses on reducing potential losses caused by uncertainty in the tail category distribution, thereby ensuring stable model performance on both head and tail categories. By fully considering the variability of data distribution, the model's generalization ability is improved, especially in the field of user behavior modeling, ensuring robust prediction performance under different data distributions.

[0067] After training with a large amount of user behavior data, this user behavior prediction model provides general representation capabilities and supports a variety of downstream tasks such as new behavior prediction, long-term behavior generation, and cross-domain adaptation.

[0068] Specifically, future behavior prediction: Future behavior prediction is a fundamental task in user behavior modeling and serves as the primary objective in the pre-training phase. Specifically, given a user's historical behavior sequence, the model needs to predict the user's next behavior. Through this task, the model learns the sequential relationships between user behaviors, capturing complex and diverse behavioral patterns, thereby improving performance in other downstream tasks. Future behavior prediction can be described as follows: in ,in Indicates the user's date , time is and location A specific event that occurred on the

[0069] New Behavior Prediction: This user behavior prediction model is able to model previously unseen behaviors using minimal user behavior data. The model can quickly learn and adapt to new behaviors with minimal data, leveraging its pre-trained knowledge, advanced feature extraction capabilities, and deep understanding of context and semantics.

[0070] In order to adapt the model faster, migrate as many parameters of the pre-trained model as possible, such as Figure 3 Specifically, in the embedding layer In the Transformer block, the model embeds the behaviors it has encountered before and initializes the embedding for the new behaviors. In the prediction layer, all parameters are transferred from the pre-trained model, thus achieving knowledge transfer and accelerating learning and generalization of unseen behaviors. In , this layer stores the score weights for each user behavior, retains the weights of previously seen behaviors, and initializes new weights for unseen behaviors.

[0071] Long-term behavior generation: This user behavior prediction model also supports the long-term generation of user behavior sequences. Leveraging a user's historical behavior, the model can generate future behaviors in an autoregressive manner, extending to a specified sequence length. By iteratively generating the next behavior and using it as input for subsequent steps, the model can model long-term user behavior trends and patterns. Long-term behavior generation can be described as follows: in, Represents the length of the generated sequence.

[0072] Cross-domain adaptation: After training on a large amount of rich, dense and high-quality user behavior data, this user behavior prediction model has the ability to adapt to cross-domains. However, unlike the basic model ChatGPT in the language field, the behavior embedding of the embodiment of the present invention may not be completely transferable, such as Figure 4 To this end, a cross-domain adaptation method suitable for this user behavior prediction model is proposed to enhance the migration ability of behavior embedding between different domains.

[0073] The embedding layer captures domain-specific representations of user behavior that are closely tied to the unique characteristics of the source domain. Since user behavior patterns can differ significantly across domains, directly transferring these embeddings can result in poor performance. On the other hand, the Transformer block learns sequential dependencies and relationships between inputs through an attention mechanism, making it more generalizable across domains. Given that user behavior often follows similar sequential patterns, the Transformer block can effectively learn abstract dependencies that can be reused in the target domain. As for the prediction layer, the MLP transforms the learned representations into outputs. The process of predicting user intent or outcomes is often similar and can therefore benefit from transfer learning. However, due to differences in the need for precise mapping, the final projection layer is not transferred.

[0074] The final proposed model achieved excellent performance as shown in Table 1, Table 2, Table 3 and Table 4: Table 1 Future behavior prediction results Table 2 New behavior prediction Table 3 Long-term behavior generation Table 4 Cross-domain prediction The training method of the user behavior prediction model provided by the present invention obtains the second historical behavior data and the third historical behavior data of the sample user, wherein the historical behavior data includes date information, time period information, location information and event information, and the third historical behavior data is the future behavior data corresponding to the second historical behavior data; extracts the historical behavior features of the sample user based on the date information, time period information, location information and event information in the second historical behavior data; inputs the historical behavior features and the third historical behavior data into the initial model for training, optimizes the hyperparameters of the initial model during the training process, and determines that the model training is completed when the accuracy of the output result of the initial model meets the conditions, thereby obtaining a user behavior prediction model. Compared with the prior art (1), which ignores the problem of uneven distribution of user behaviors, the method is particularly deficient in processing long-tail user intentions. These long-tail intentions refer to rare or uncommon behaviors of a small number of users. Due to data sparsity, modeling and predicting these behaviors become extremely difficult. The lack of data limits the model's ability to learn uncommon user intentions, making it difficult to accurately capture the complex patterns therein, resulting in poor performance when generalizing to other areas of user behavior; (2) The model is sensitive to noise in the behavior sequence and is easily disturbed by accidental behavior or data anomalies, which leads to unstable performance of the model and affects its application effect in real scenarios; (3) Existing patents often focus on a certain area of ​​user behavior, and the user behavior pattern is relatively single, which often cannot fit the complex and diverse behavior patterns of users at all times of the day and night, and it is difficult to flexibly characterize the user's multiple behavioral preferences and situational changes, thereby affecting the accurate prediction of user behavior; (4) The existing model is not scalable enough, especially when facing large-scale user behavior data. It is impossible to improve the model's capabilities by increasing model parameters or simply stacking, which limits its practicality in big data scenarios. This method makes user behavior prediction more fair through a novel pre-training paradigm, improves the effect of long-tail behavior, and improves the generalization ability of the model. The model input includes features such as week, timestamp, location, and historical events, and each feature is processed through the corresponding embedding layer. These embeddings are then combined and passed through a Transformer block to capture the relationships between different features. Finally, the prediction layer leverages this learned information to predict future user behavior. To ensure fair modeling for each user and improve the model's generalization capabilities, a pre-training paradigm is introduced that minimizes potential losses from the uncertain tail class distribution. This approach ensures that the model effectively performs on both head and tail behaviors, thereby improving the model's fairness and robustness.

[0075] The user behavior prediction device provided by the present invention is described below. The user behavior prediction device described below and the user behavior prediction method described above can be referenced to each other.

[0076] Figure 7 Schematic diagram of the structure of the user behavior prediction device provided by the present invention, which specifically includes: The acquisition module 701 is used to acquire the first historical behavior data of the target user and the target prediction task. Detailed descriptions can be found in the corresponding descriptions of the above method embodiments, which will not be repeated here.

[0077] Prediction module 702 is configured to input the first historical behavior data into a pre-trained user behavior prediction model based on the target prediction task, and predict the target user's future target behavior using the user behavior prediction model. The future target behavior includes historical behaviors, historical behaviors, and long-term behaviors. For detailed descriptions, please refer to the corresponding descriptions of the above method embodiments and will not be repeated here.

[0078] The model training module 703 is used to obtain the second historical behavior data and the third historical behavior data of the sample user, wherein the historical behavior data includes date information, time period information, location information and event information, and the third historical behavior data is the future behavior data corresponding to the second historical behavior data; extract the historical behavior characteristics of the sample user based on the date information, time period information, location information and event information in the second historical behavior data; input the historical behavior characteristics and the third historical behavior data into the initial model for training, and optimize the hyperparameters of the initial model during the training process until the accuracy of the output result of the initial model meets the conditions, and determine that the model training is completed to obtain the user behavior prediction model. For detailed descriptions, please refer to the relevant descriptions corresponding to the above method embodiments, which will not be repeated here.

[0079] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8As shown, the electronic device may include: a processor 810 , a communication interface 820 , a memory 830 and a communication bus 840 , wherein the processor 810 , the communication interface 820 and the memory 830 communicate with each other via the communication bus 840 . The processor 810 can call the logic instructions in the memory 830 to execute the user behavior prediction method, which includes: obtaining the first historical behavior data and target prediction task of the target user; inputting the first historical behavior data into the pre-trained user behavior prediction model based on the target prediction task, and predicting the future target behavior of the target user through the user behavior prediction model, wherein the future target behavior includes historical behaviors, historical behaviors that have not occurred, and long-term behaviors; wherein the user behavior prediction model is trained based on the following steps: obtaining the second historical behavior data and third historical behavior data of the sample user, wherein the historical behavior data includes date information, time period information, location information and event information, and the third historical behavior data is the future behavior data corresponding to the second historical behavior data; extracting the historical behavior features of the sample user based on the date information, time period information, location information and event information in the second historical behavior data; inputting the historical behavior features and the third historical behavior data into the initial model for training, and optimizing the hyperparameters of the initial model during the training process until the accuracy of the output result of the initial model meets the conditions, determining that the model training is completed, and obtaining the user behavior prediction model.

[0080] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0081] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the user behavior prediction method provided by the above methods, which includes: obtaining the first historical behavior data and target prediction task of the target user; inputting the first historical behavior data into a pre-trained user behavior prediction model based on the target prediction task, and predicting the future target behavior of the target user through the user behavior prediction model, wherein the future target behavior includes historical behaviors, historical behaviors that have not occurred, and long-term behaviors; wherein the user behavior prediction model is based on The training is performed in the following steps: obtaining the second historical behavior data and the third historical behavior data of the sample user, wherein the historical behavior data includes date information, time period information, location information and event information, and the third historical behavior data is the future behavior data corresponding to the second historical behavior data; extracting the historical behavior characteristics of the sample user based on the date information, time period information, location information and event information in the second historical behavior data; inputting the historical behavior characteristics and the third historical behavior data into the initial model for training, optimizing the hyperparameters of the initial model during the training process, and determining that the model training is completed when the accuracy of the output result of the initial model meets the conditions, and obtaining a user behavior prediction model.

[0082] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the user behavior prediction method provided by the above-mentioned methods, the method comprising: obtaining first historical behavior data and a target prediction task of a target user; inputting the first historical behavior data into a pre-trained user behavior prediction model based on the target prediction task, and predicting the future target behavior of the target user through the user behavior prediction model, wherein the future target behavior includes behaviors that have occurred in the past, behaviors that have not occurred in the past, and long-term behaviors; wherein the user behavior prediction model is trained based on the following steps: obtaining second historical behavior data and third historical behavior data of a sample user, wherein the historical behavior data includes date information, time period information, location information, and event information, and the third historical behavior data is the future behavior data corresponding to the second historical behavior data; extracting the historical behavior features of the sample user based on the date information, time period information, location information, and event information in the second historical behavior data; inputting the historical behavior features and the third historical behavior data into an initial model for training, optimizing the hyperparameters of the initial model during the training process, and determining that the model training is completed when the accuracy of the output result of the initial model meets the conditions, thereby obtaining a user behavior prediction model.

[0083] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0084] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or alternatively, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes instructions for enabling a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or portions thereof.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A user behavior prediction method, characterized in that: include: Obtain the target user's first historical behavior data and target prediction tasks; Inputting the first historical behavior data into a pre-trained user behavior prediction model based on the target prediction task, and predicting the target user's future target behavior through the user behavior prediction model, wherein the future target behavior includes historical behaviors, historical behaviors that have not occurred, and long-term behaviors; The user behavior prediction model is trained based on the following steps: Obtaining second historical behavior data and third historical behavior data of the sample user, wherein the historical behavior data includes date information, time period information, location information, and event information, and the third historical behavior data is future behavior data corresponding to the second historical behavior data; extracting the historical behavior features of the sample user based on the date information, time period information, location information, and event information in the second historical behavior data; The historical behavior characteristics and the third historical behavior data are input into the initial model for training. During the training process, the hyperparameters of the initial model are optimized until the accuracy of the output result of the initial model meets the conditions. The model training is determined to be completed and the user behavior prediction model is obtained.

2. The method according to claim 1, characterized in that The user behavior prediction model includes a day embedding layer, a timestamp embedding layer, a location embedding layer and an event embedding layer, and the historical behavior features include date features, time features, location features and event features; The extracting the historical behavior features of the sample user based on the date information, time period information, location information, and event information in the second historical behavior data includes: The date information is converted into a corresponding date feature embedding vector with the corresponding week ID through the day embedding layer; The time period information is divided into multiple time periods according to the time stamp embedding layer, and the time period ID corresponding to each time period is converted into a corresponding time feature embedding vector; Converting the geographic location ID corresponding to the location information into a corresponding location feature embedding vector through the location embedding layer; The event features of the event information are extracted through the event embedding layer, and the event features are converted into event feature embedding vectors.

3. The method according to claim 2, characterized in that Inputting the historical behavior features and the third historical behavior data into an initial model for training, and optimizing hyperparameters of the initial model during the training process, includes: Extracting the dependency relationship among the date feature embedding vector, the time feature embedding vector, the location feature embedding vector, and the event feature embedding vector through an attention mechanism, and obtaining a mapping function of the historical event sequence corresponding to the second historical behavior data; Training the initial model based on the mapping function and the third historical behavior data; The hyperparameters of the initial model are set, and during the model training process, the hyperparameters are optimized based on the results of each round of training.

4. The method according to claim 3, characterized in that The user behavior prediction model also includes a prediction layer; The training of the initial model based on the mapping function and the third historical behavior data includes: Using a multilayer perceptron to learn the mapping function to obtain an output result; The output result is compared with the third historical behavior data. When the accuracy of the output result is greater than a threshold, it is determined that the initial model training is completed, and the trained initial model is used as a user behavior prediction model.

5. The method according to claim 3, characterized in that Setting the hyperparameters of the initial model and optimizing the hyperparameters based on each round of training results during the model training process include: Based on the preset model training mechanism and the results of each round of training, the hyperparameters are optimized using the distributed robust optimization method.

6. The method according to claim 1, characterized in that Inputting the first historical behavior data into a pre-trained user behavior prediction model based on the target prediction task, and predicting the future target behavior of the target user by using the user behavior prediction model, includes: If the target prediction task is a future behavior prediction task, the first historical behavior data is input into a pre-trained user behavior prediction model, and the future behavior of the target user is predicted by the user behavior prediction model, where the future behavior is a historical behavior; If the target prediction task is a new behavior prediction task, the user behavior prediction model is trained again based on behavior data that has not occurred in the past; Inputting the first historical behavior data into the twice-trained user behavior prediction model, and predicting the target user's future new behavior using the twice-trained user behavior prediction model; If the target prediction task is a long-term behavior generation task, the first historical behavior data is input into a pre-trained user behavior prediction model, and future behaviors are generated in an autoregressive manner through the user behavior prediction model and extended to a preset sequence length.

7. A user behavior prediction device, characterized in that: include: An acquisition module, used to obtain the first historical behavior data and target prediction task of the target user; A prediction module, configured to input the first historical behavior data into a pre-trained user behavior prediction model based on the target prediction task, and predict the target user's future target behavior using the user behavior prediction model, wherein the future target behavior includes historical behaviors, historical behaviors that have not occurred, and long-term behaviors; A model training module is used to obtain the second historical behavior data and third historical behavior data of the sample user, wherein the historical behavior data includes date information, time period information, location information and event information, and the third historical behavior data is the future behavior data corresponding to the second historical behavior data; based on the date information, time period information, location information and event information in the second historical behavior data, the historical behavior characteristics of the sample user are extracted; the historical behavior characteristics and the third historical behavior data are input into the initial model for training, and the hyperparameters of the initial model are optimized during the training process until the accuracy of the output result of the initial model meets the conditions, the model training is determined to be completed, and a user behavior prediction model is obtained.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the user behavior prediction method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the user behavior prediction method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the user behavior prediction method according to any one of claims 1 to 6 is implemented.

Images