User behavior prediction method and device, electronic equipment and medium
By converting user behavior data into natural language formats and fine-tuning of two-stage large language models, the problems of low utilization of behavior prediction data and inaccurate long-tail behavior prediction are solved, and efficient utilization of behavior data and accurate prediction of long-tail behavior are achieved.
Patent Information
- Application Number
- CN202510493681.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-18
- Publication Date
- 2025-08-29
AI Technical Summary
In the prior art, the utilization rate of behavior prediction data is low and the prediction of low probability behavior is inaccurate, especially the prediction of long-tail distribution behavior is inaccurate, making it difficult to achieve better prediction performance in any scenario.
By converting user behavior data into natural language formats and using preset prompt words to train the initial model, a progressive two-stage large language model fine-tuning strategy is adopted, first fine-tuning the anchor behavior, and then selecting samples based on the difficulty coefficient to fine-tuning the long-tail behavior data, improving the model's prediction ability of long-tail behavior.
It effectively improves the utilization efficiency of behavioral data and the prediction accuracy of long-tail behavior, ensuring good results in any type of behavior prediction.
Smart Images
Figure CN120561641A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of big data and user behavior prediction, and in particular to a user behavior prediction method, device, electronic device and medium. Background Art
[0002] Most people lead regular lives driven by behavioral habits, but they also exhibit short bursts of activity influenced by specific circumstances. The ability to predict a user's next behavior is crucial for modern intelligent assistant services, which span a wide range of applications from web platforms to smart devices.
[0003] With the accumulation of empirical data on user behavior, deep learning-based methods have replaced traditional rule-based approaches and become the mainstream solution in this field. Transformer-based models excel at capturing transition patterns between sequential behaviors. However, the key to accurately predicting user behavior lies in understanding the underlying intentions behind observed behaviors, rather than simply modeling the behavior sequence. For example, when a user leaves home later than usual for work on a weekday due to unexpected circumstances, they may choose to take a taxi instead of public transportation to arrive on time to complete their attendance check-in. However, deep learning-based methods often fail to achieve good predictions in such scenarios. Furthermore, deep learning-based methods require a large amount of behavioral data to train the model, which places certain demands on the data volume. In some cases, only limited behavioral data is available, which places high demands on the model's data utilization efficiency. Existing technologies primarily focus on extracting feature information from user behavior data and then using this extracted feature information to predict user behavior through machine learning methods. This places certain data requirements on the user, but fails to address the efficiency of actual behavioral data samples and overlooks the challenges and significance of predicting long-tail user behavior.
[0004] In this context, large language models (LLMs) have become an ideal choice for behavior prediction. These pre-trained large language models inherently store knowledge about human behavior from the large text corpora on which they are trained. This makes it possible to predict emergent behavior in specific scenarios and significantly reduces the reliance on large-scale behavioral data, enabling them to achieve certain results after training on a small amount of behavioral data. To adapt LLMs to behavior prediction, existing work typically converts behavioral sequence data into text format and fine-tunes the LLM to predict user behavior tokens. Early attempts proposed combining general instruction tuning for conversational data with lightweight tuning for behavioral data. Subsequent work further integrated the encoding embeddings of traditional models into text instructions to facilitate LLM's understanding of user behavior. Although some progress has been made in predicting human behavior using large language models, it still does not address the problem of inaccurate predictions for long-tail distributed behaviors. Specifically, a few behaviors corresponding to a few types with higher frequency dominate most of the user's daily life, while other behaviors occur much less frequently. Both deep learning methods and existing LLM methods find it difficult to effectively capture such long-tail distributed behaviors. In real life, the types of long-tail behaviors cannot be underestimated, and the prediction accuracy of long-tail behaviors is often a key factor affecting user service satisfaction.
[0005] In summary, how to effectively and accurately predict human behavior mainly faces the following challenges: (1) How to effectively improve the utilization efficiency of behavioral data and reduce the dependence on large-scale behavioral data so that it can achieve a certain prediction performance in any scenario; (2) How to effectively improve the prediction effect of long-tail behavior so that the model has a certain effect on any type of behavior prediction. Summary of the Invention
[0006] The present invention provides a user behavior prediction method, device, electronic device and medium to address the defects of low utilization rate of behavior prediction data and inaccurate prediction of behaviors with low probability in the prior art, thereby achieving efficient utilization of behavior data and accurate prediction of long-tail intention behaviors.
[0007] The present invention provides a user behavior prediction method, comprising: Obtain the target user's first historical behavior data; Inputting the first historical behavior data into a user behavior prediction model, and predicting the target user's future behavior and corresponding behavior type using the user behavior prediction model, wherein the behavior type includes anchor behavior and long-tail behavior, wherein the anchor behavior includes behavior for which the target type's behavior data accounts for a greater than or equal to a preset percentage of the sum of all types of behavior data, and the long-tail behavior includes behavior other than the anchor behavior; The user behavior prediction model is trained based on the following steps: Obtaining multiple second historical behavior data of sample users; Converting each of the second historical behavior data into a behavior data text in a natural language format to obtain a plurality of second historical behavior data texts; The initial model is trained based on the preset prompt words and the multiple second historical behavior data texts. When the accuracy of the output result of the initial model meets the conditions, the initial model is fine-tuned based on the anchor point behavior and the equilibrium behavior type data to obtain the user behavior prediction model.
[0008] In one possible implementation, the method further includes: The step of converting each of the second historical behavior data into a behavior data text in a natural language format to obtain a plurality of second historical behavior data texts includes: According to the preset digital ID mapping table, the digital ID string of each second historical behavior data is converted into a behavior data text in natural language format to obtain multiple second historical behavior data texts, wherein the preset digital ID mapping table includes multiple digital IDs and natural language description information corresponding to each digital ID.
[0009] In one possible implementation, the method further includes: Generate prompt words based on prompt word design rules and the current model training task to obtain preset prompt words, wherein the prompt words are used by the initial model to understand the model training task; Arranging the plurality of second historical behavior data texts in chronological order according to the time corresponding to each second historical behavior data text to obtain a second historical behavior data text sequence; The second historical behavior data text sequence is input into the initial model for training, and the learning direction of the initial model is controlled by the preset prompt words.
[0010] In one possible implementation, the method further includes: During the initial model training process, the first N second historical behaviors of the second historical behavior data text sequence are used as model input data, and the N+1 second historical behavior is used as model output data. When the accuracy of the model output data meets a threshold, the initial model training is determined to be complete, where N>1 and N is less than the data volume of the second historical behavior data text sequence; The trained initial model is fine-tuned based on anchor point behavior and balanced behavior type data to obtain a user behavior prediction model.
[0011] In one possible implementation, the method further includes: Fine-tune the trained initial model using a preset anchor behavior dataset to obtain a first fine-tuned model; According to the sample selection strategy of the model prediction difficulty, a behavior prediction set pair with balanced behavior types is screened out from all behavior data sets, and the first fine-tuning model is fine-tuned based on the behavior prediction set to obtain a user behavior prediction model.
[0012] In one possible implementation, the method further includes: During the initial model training process, an autoregressive function is used to optimize and fine-tune the initial model.
[0013] The present invention also provides a user behavior prediction device, comprising the following modules: An acquisition module, used to acquire the first historical behavior data of the target user; a prediction module, configured to input the first historical behavior data into a user behavior prediction model, and predict the target user's future behavior and corresponding behavior type using the user behavior prediction model, wherein the behavior type includes anchor behavior and long-tail behavior, wherein the anchor behavior includes behavior for which the target type's behavior data accounts for a greater than or equal to a preset percentage of the sum of all types of behavior data, and the long-tail behavior includes behavior other than the anchor behavior; The model training module is used to obtain multiple second historical behavior data of sample users; convert each second historical behavior data into a behavior data text in a natural language format to obtain multiple second historical behavior data texts; train the initial model based on preset prompt words and the multiple second historical behavior data texts, and when the accuracy of the output result of the initial model meets the conditions, fine-tune the initial model based on the anchor point behavior and the balanced behavior data based on the difficulty coefficient to obtain the user behavior prediction model.
[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the user behavior prediction method as described above is implemented.
[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the user behavior prediction methods described above.
[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the user behavior prediction methods described above.
[0017] The user behavior prediction method, device, electronic device and medium provided by the present invention predict the future behavior of the target user and the corresponding behavior type by inputting the first historical behavior data into the user behavior prediction model, wherein the behavior type includes anchor behavior and long-tail behavior, and the anchor behavior includes the behavior whose target type behavior data accounts for a greater than or equal to a preset percentage of the sum of all types of behavior data, and the long-tail behavior includes the behavior other than the anchor behavior; wherein the user behavior prediction model is trained based on the following steps: obtaining multiple second historical behavior data of a sample user; converting each second historical behavior data into a behavior data text in a natural language format to obtain multiple second historical behavior data texts; training the initial model based on preset prompt words and the multiple second historical behavior data texts, and when the accuracy of the output result of the initial model meets the conditions, fine-tuning the initial model based on the anchor behavior and the balanced behavior type data to obtain the user behavior prediction model. Compared with the existing behavior prediction methods that have low data utilization and inaccurate predictions for behaviors with low probability, this solution effectively utilizes the general behavior knowledge stored in the pre-trained large language model to perform behavior prediction through a progressive two-stage large language model fine-tuning strategy, thereby improving the efficiency of behavior data utilization and the accuracy of predictions for behaviors with low probability. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 It is a flow chart of the user behavior prediction method provided by the present invention.
[0020] Figure 2 This is one of the flow charts of the training method of the user behavior prediction model provided by the present invention.
[0021] Figure 3 This is the second flow chart of the training method of the user behavior prediction model provided by the present invention.
[0022] Figure 4 This is a schematic diagram of the data sample input-output process provided by the present invention.
[0023] Figure 5 It is a structural diagram of the user behavior prediction device provided by the present invention.
[0024] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0025] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.
[0026] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below with reference to the accompanying drawings. The embodiments do not limit the embodiments of the present invention.
[0027] Figure 1 is a flow chart of the user behavior prediction method provided by the present invention, such as Figure 1 As shown, the method includes the following: S11. Obtain the first historical behavior data of the target user.
[0028] The embodiments of the present invention are primarily used in scenarios where users' future behaviors are predicted. In particular, they can accurately predict behaviors that have a low probability of occurring. For example, if a user leaves home later than usual for work on a weekday due to unexpected reasons, they may choose to take a taxi to the company instead of taking public transportation because they need to arrive at the company on time to complete their attendance check-in.
[0029] Historical user behavior data typically comes from multiple sources, including but not limited to purchase records, browsing history, and click activity on e-commerce platforms, social media interaction records (such as likes, comments, and shares), mobile app usage behavior, and phone usage logs. This data can reflect user behavior patterns and preferences in different scenarios. Historical user behavior data can be divided into explicit feedback (such as purchase behavior and ratings) and implicit feedback (such as browsing and dwell time). Explicit feedback directly reflects user preferences, while implicit feedback requires analysis to uncover its underlying meaning.
[0030] Optionally, after acquiring the data, it needs to be cleaned and normalized. For example, duplicate records can be removed, missing values can be filled, and data can be normalized. Furthermore, feature engineering can be used to extract more meaningful features, such as user activity and preferred categories.
[0031] Optionally, by analyzing historical behavior data, you can build user profiles, including demographic characteristics (such as age and gender), behavioral preferences (such as purchase frequency and browsing categories), and interest tags. This profile information helps better understand user characteristics and provides a basis for subsequent behavior prediction.
[0032] S12: Input the first historical behavior data into a user behavior prediction model, and predict the future behavior of the target user and the corresponding behavior type through the user behavior prediction model.
[0033] User behavior prediction models can be built using a variety of algorithms, including logistic regression, random forests, and deep learning models (such as neural networks). These models learn from historical user behavior data to identify correlations between behavioral patterns and future behavior. User behavior prediction models are widely used in areas such as personalized recommendations, precision marketing, and user retention. For example, e-commerce platforms can provide personalized product recommendations based on predictions, improving user satisfaction and purchase conversion rates. The detailed model training process is not detailed here.
[0034] Furthermore, the pre-processed and feature-extracted user historical behavior data is fed into a pre-trained model. Based on this input data, the model calculates the user's likely future behaviors, their probabilities, and their types. For example, it predicts whether the user will purchase a product, click on an ad, or browse a page. Behavior types include anchor behaviors and long-tail behaviors. Anchor behaviors are behaviors whose data for a particular type accounts for a predetermined percentage or more of the total data for all other types of behaviors. Long-tail behaviors are behaviors other than anchor behaviors.
[0035] The user behavior prediction method provided by the present invention inputs the first historical behavior data into a user behavior prediction model, and predicts the future behavior of the target user and the corresponding behavior type through the user behavior prediction model, wherein the behavior type includes anchor behavior and long-tail behavior, the anchor behavior includes the behavior of which the target type of behavior data accounts for a sum of all types of behavior data greater than or equal to a preset percentage, and the long-tail behavior includes the behavior other than the anchor behavior; wherein the user behavior prediction model is trained based on the following steps: obtaining multiple second historical behavior data of a sample user; converting each second historical behavior data into a behavior data text in a natural language format to obtain multiple second historical behavior data texts; training the initial model based on preset prompt words and the multiple second historical behavior data texts, and when the accuracy of the output result of the initial model meets the conditions, fine-tuning the initial model based on the anchor behavior and the balanced behavior type data to obtain the user behavior prediction model. Compared with the existing behavior prediction methods that have low data utilization and inaccurate predictions for behaviors with low probability, this method effectively utilizes the general behavior knowledge stored in the pre-trained large language model to perform behavior prediction through a progressive two-stage large language model fine-tuning strategy, thereby improving the efficiency of behavior data utilization and the accuracy of predictions for behaviors with low probability.
[0036] Figure 2 This is one of the flow charts of the training method of the user behavior prediction model provided by the present invention, such as Figure 2 As shown, the method includes the following: S21. Obtain multiple second historical behavior data of the sample user.
[0037] The embodiment of the present invention describes in detail the training method of the user behavior prediction model, combined with Figure 3 As shown, first, historical behavioral data of sample users typically comes from multiple channels, including but not limited to purchase records, browsing history, and click behavior on e-commerce platforms, interaction records on social media (such as likes, comments, and shares), usage behavior in mobile apps, and phone usage logs. This data can reflect user behavior patterns and preferences in different scenarios.
[0038] S22: Convert each of the second historical behavior data into a behavior data text in a natural language format to obtain a plurality of second historical behavior data texts.
[0039] Each second historical behavior data includes a digital ID string. The digital ID string of each second historical behavior data is converted into a behavior data text in a natural language format according to a preset digital ID mapping table to obtain multiple second historical behavior data texts.
[0040] For example, the preset digital ID mapping table records 1 as "home", 2 as "online shopping", 3 as "video", 4 as "sports", etc. Each second historical behavior data is replaced with a text description, such as (1, 16 - home, sports), ... (1, 2 - home, online shopping).
[0041] S23. The initial model is trained based on the preset prompt words and the multiple second historical behavior data texts. When the accuracy of the output result of the initial model meets the conditions, the initial model is fine-tuned based on the anchor behavior and the balanced behavior data based on the difficulty coefficient to obtain a user behavior prediction model.
[0042] Based on the prompt word design rules and the current model training task, a prompt word is generated to obtain a preset prompt word. This prompt word is used for the initial model understanding model training task. Furthermore, the multiple second historical behavior data texts are arranged in chronological order based on the time corresponding to each second historical behavior data text to obtain a second historical behavior data text sequence. The second historical behavior data text sequence is input into the initial model for training, and the learning direction of the initial model is controlled by the preset prompt word.
[0043] Specifically, a pure text prompt word design is used to allow user behavior data to be converted into training data suitable for large language model instruction tuning. The prompt word consists of the following five parts: 1. Task definition: A brief description of the next action prediction task, for example, predicting the next action of a candidate user.
[0044] 2. Role-playing guidance: This defines the roles and tasks that the large language model is expected to assume and complete. For example, if you are a smart user’s mobile phone assistant, you can infer the user’s mobile phone behavior preferences based on the user’s historical behavior history.
[0045] 3. Historical behavior sequence (model input): Replace each historical behavior data with a text description, such as (1, 16-home, exercise), ... (1, 2-home, online shopping).
[0046] 4. Candidate set (model input): The set of all possible behaviors that the large language model can predict, limiting the range of selectable behaviors, such as checking the weather, music, ..., cycling, reading.
[0047] 5. Next action (model output): The ground truth label that the large language model wants to predict, for example, game.
[0048] The prediction task of the embodiment of the present invention is to predict the next behavior in the future based on historical sequence information, such as Figure 4 As shown. Given a user's historical sequence , predicting users’ future behavior , is a set of all behavior types, where events , indicating the type Events at time Time and location Occurrence, time Contains two elements, representing the day of the week and the hour of the day the event occurs. Therefore, the prediction task can be expressed as: Given a behavior dataset , the goal is to train a model that can predict the next behavior ,Right now .
[0049] According to the above prompt word design description, the actual data set of the model will be converted into a text instruction format data set ,in Represent the text formats corresponding to the input and output data respectively, and the prediction task can be expressed as During the initial model training process, an autoregressive function training objective can be used to fine-tune the large language model. The optimization function can be expressed as: in, represents the learnable parameters of the large language model, express The corresponding token length, Indicates the first tokens, Indicates all All previous tokens.
[0050] Furthermore, during the initial model training process, the first N second historical behaviors in the second historical behavior data text sequence are used as model input data, and the N+1th second historical behavior is used as model output data. When the accuracy of the model output data meets the threshold, the initial model training is determined to be completed; the trained initial data is fine-tuned based on the anchor behavior based on multi-task enhancement and the balanced behavior data based on the difficulty coefficient to obtain a user behavior prediction model.
[0051] Specifically, based on the multi-task enhanced anchor behavior fine-tuning stage, the large language model is fine-tuned using anchor behavior data, focusing on user behavior prediction while retaining its inherent rich long-tail behavior knowledge learned from pre-training.
[0052] Definition of anchor behavior and long-tail behavior: If a certain type of behavior data accounts for 1% or more of the sum of all types of behavior data, then this type of behavior is defined as anchor behavior; otherwise, it is defined as long-tail behavior.
[0053] In the equilibrium behavior data fine-tuning stage based on the difficulty coefficient, a sample selection strategy based on the model prediction difficulty was designed. Based on this, a small and precise behavior prediction set of various types of equilibrium was selected from the behavior data set. By fine-tuning on this data set, the model's prediction ability for long-tail behavior was enhanced.
[0054] Specifically, the fine-tuning data of instructions in natural language format is divided into two parts: anchor behavior instruction fine-tuning data and long-tail behavior instruction fine-tuning data. At the current stage, only the anchor behavior instruction fine-tuning dataset is used. Fine-tune the large language model. Regarding anchor behavior data, according to statistics, it accounts for more than 90% of all user behavior data, and the behavior types corresponding to anchor behaviors can basically cover the user's usage needs, because we believe that anchor behavior data already represents the core patterns of users' daily lives. Therefore, after fine-tuning the large language model using anchor behaviors, the large language model has a preliminary understanding of the potential patterns of user behavior, so that it can accurately predict the next anchor behavior. In addition, because the large language model has captured general behavioral knowledge from the pre-training corpus, fine-tuning of anchor behaviors makes it easier to generalize to other unseen but semantically similar long-tail behaviors. Compared with fine-tuning using the entire behavioral dataset, this method helps prevent the large language model from being biased towards predicting some anchor behavior types that account for a higher proportion in the data.
[0055] At the same time, in order to further improve the ability of the large language model to generalize from predicting anchor behaviors to long-tail behaviors, the embodiment of the present invention also fine-tunes the model in the form of auxiliary tasks. In addition to learning to predict the next behavior, it is also intended to enable the model to maintain its ability to solve general tasks. To achieve this, another dataset derived from a general corpus is also used. Fine-tune the large language model, From daily conversations between users and ChatGPT. This effectively integrates the auxiliary task of conversation generation and the main task of behavior prediction. The size of the auxiliary task is used to control the influence of the auxiliary task, that is, to use the In practice, We filter out long dialogues in , to ensure that prompts from both tasks are comparable in length.
[0056] Since the large language model trained in the first stage has served as a good zero-probability predictor for long-tail behaviors, we believe that a small amount of instruction fine-tuning data covering all behavior types is sufficient to build a high-performance user behavior predictor. Therefore, in this stage, the tail behavior dataset is reintroduced. , and compare them with the dataset Combined, they create a small, refined, and class-balanced fine-tuning dataset. However, to achieve this goal, the quality and information content of the selected samples play a crucial role. To this end, a difficulty-based data selection method was developed to find high-quality samples based on two dimensions (prediction difficulty and class diversity), allowing for efficient fine-tuning of large language models using a small amount of data.
[0057] A simple way to measure the difficulty of sample prediction is to use the intermediate model to score the sample. Generally speaking, it is more difficult to predict the wrong sample than the correct sample. In the implementation, the fine-tuned large language model from the first stage (the large language model after fine-tuning the anchor behavior) is directly used as the sample scorer, and the intermediate model is used to score the sample. and Filter out incorrectly predicted samples for each behavior type. On the other hand, following the principle of contrastive learning, it is believed that distinguishing long-tail behaviors from anchor behaviors, or anchor behaviors from long-tail behaviors, is more difficult than distinguishing behaviors of the same type. Therefore, we select samples in which the fine-tuned large language model in the first stage made incorrect predictions, and whose incorrect labels belong to the same general category (anchor behaviors or long-tail behaviors) as the true labels to form the small, precise, and balanced fine-tuning dataset described above. A fixed number of samples are selected for each behavior type to maintain a balanced number of behaviors across categories.
[0058] By adopting a difficulty-based data selection strategy, we can encourage large language models to capture more refined behavioral knowledge in a sample-efficient manner and significantly improve their prediction accuracy of cross-tail behaviors without affecting the anchor behavior.
[0059] The present invention evaluated the user behavior prediction model on two real user behavior datasets, the details of which are as follows: Honor Behavior Dataset: This large-scale dataset is derived from Honor mobile phone usage logs. When users interact with their phones, various types of logs are generated. These logs are desensitized and reported with the user's consent. 37 types of daily behaviors are reliably extracted from the raw logs, covering a wide range of life scenarios, including activities related to learning, work, entertainment, and leisure. The dataset spans from March 1, 2024, to April 29, 2024, and consists of over 50 million behavioral events from 24,133 anonymous users. The dataset is preprocessed and samples are constructed in the format of "(last 20 historical user behaviors, next user behavior)". Because the goal is to fine-tune a large language model and leverage the pre-trained knowledge it contains to assist in behavior prediction, only a randomly selected subset of 200,000 is used for the experiment.
[0060] App Usage Behavior Dataset: This dataset contains the app usage data of 1,753 anonymous users over a week. Given the significant functional overlap between apps, the dataset was further processed by merging apps with similar purposes. For example, TikTok and Kuaishou were grouped into the "watch videos" category. After similar preprocessing as the Honor Behavior Dataset, the dataset ultimately consists of 24 categories of behavior.
[0061] LLama3.1-8B was selected as the large language model for the experiment. In order to ensure the flexibility of model testing, three different instruction formats were designed in the embodiment of the present invention, which were randomly sampled during training and testing. This experiment used the AdamW optimizer with a cosine annealing learning rate schedule, and set the warm-up ratio to 0.1. The maximum learning rate of cosine annealing was set to 1e-4, while the minimum learning rate of both was set to 1e-6. LoRA fine-tuning and parallel training acceleration were performed using the open source large language model instruction fine-tuning library llama-factory. All experiments were performed with a maximum of 8 training epochs and 8 batch sizes. The best performing model was selected for testing on the validation set. For each experiment, at least 5 experiments with different random seeds were performed, and the average value was reported. The experimental results are shown in Table 1.
[0062] Table 1 The user behavior prediction method provided by the present invention inputs the first historical behavior data into a user behavior prediction model, and predicts the future behavior of the target user and the corresponding behavior type through the user behavior prediction model, wherein the behavior type includes anchor behavior and long-tail behavior, the anchor behavior includes the behavior of which the target type of behavior data accounts for a sum of all types of behavior data greater than or equal to a preset percentage, and the long-tail behavior includes the behavior other than the anchor behavior; wherein the user behavior prediction model is trained based on the following steps: obtaining multiple second historical behavior data of a sample user; converting each second historical behavior data into a behavior data text in a natural language format to obtain multiple second historical behavior data texts; training the initial model based on preset prompt words and the multiple second historical behavior data texts, and when the accuracy of the output result of the initial model meets the conditions, fine-tuning the initial model based on the anchor behavior and the balanced behavior type data to obtain the user behavior prediction model. Compared to existing behavior prediction methods, which suffer from low data utilization and inaccurate predictions for low-probability behaviors, this method employs a progressive, two-stage large language model fine-tuning strategy. This strategy first textualizes the raw behavior data and converts it into corresponding prompt word data. It then fine-tunes the model in the first stage by filtering out anchor behavior data from the dataset and mixing it with auxiliary training data. Subsequently, a sample selection strategy based on model prediction difficulty is used to select a small, well-balanced behavior dataset, which is then used to further fine-tune the model trained in the first stage. This approach enables the model to achieve good prediction results for both anchor and long-tail behaviors.
[0063] The user behavior prediction device provided by the present invention is described below. The user behavior prediction device described below and the user behavior prediction method described above can be referenced to each other.
[0064] Figure 5 Schematic diagram of the structure of the user behavior prediction device provided by the present invention, which specifically includes: The acquisition module 501 is used to acquire the first historical behavior data of the target user. Detailed descriptions can be found in the corresponding descriptions of the above method embodiments, which will not be repeated here.
[0065] Prediction module 502 is configured to input the first historical behavior data into a user behavior prediction model and predict the target user's future behavior and corresponding behavior type using the user behavior prediction model. The behavior types include anchor behaviors and long-tail behaviors. Anchor behaviors include behaviors for which the target type's behavior data accounts for a greater than or equal to a preset percentage of the sum of all behavior types, and long-tail behaviors include behaviors other than the anchor behaviors. For detailed descriptions, please refer to the corresponding descriptions of the aforementioned method embodiments and will not be repeated here.
[0066] Model training module 503 is configured to obtain multiple second historical behavior data of a sample user; convert each second historical behavior data into a behavior data text in natural language format to obtain multiple second historical behavior data texts; train an initial model based on preset prompt words and the multiple second historical behavior data texts; and when the accuracy of the output result of the initial model meets the requirements, fine-tune the initial model based on the anchor behavior and the balanced behavior data based on the difficulty coefficient to obtain the user behavior prediction model. For detailed descriptions, please refer to the corresponding descriptions of the above-mentioned method embodiments and will not be repeated here.
[0067] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6 As shown, the electronic device may include: a processor 810 , a communication interface 820 , a memory 830 and a communication bus 840 , wherein the processor 810 , the communication interface 820 and the memory 830 communicate with each other via the communication bus 840 . The processor 810 can call the logic instructions in the memory 830 to execute the user behavior prediction method, which includes: inputting the first historical behavior data into the user behavior prediction model, and predicting the future behavior of the target user and the corresponding behavior type through the user behavior prediction model, wherein the behavior type includes anchor behavior and long-tail behavior, and the anchor behavior includes the behavior of the target type whose behavior data accounts for a sum of all types of behavior data greater than or equal to a preset percentage, and the long-tail behavior includes behavior other than the anchor behavior; wherein the user behavior prediction model is trained based on the following steps: obtaining multiple second historical behavior data of the sample user; converting each second historical behavior data into a behavior data text in a natural language format to obtain multiple second historical behavior data texts; training the initial model based on preset prompt words and the multiple second historical behavior data texts, and when the accuracy of the output result of the initial model meets the conditions, fine-tuning the initial model based on the anchor behavior and the balanced behavior type data to obtain the user behavior prediction model.
[0068] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.
[0069] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the user behavior prediction method provided by the above methods, the method including: inputting the first historical behavior data into a user behavior prediction model, and predicting the future behavior and corresponding behavior type of the target user through the user behavior prediction model, wherein the behavior type includes anchor behavior and long-tail behavior, the anchor behavior includes behavior in which the target type behavior data accounts for a sum of all types of behavior data greater than or equal to a preset percentage, and the long-tail behavior includes behavior other than the anchor behavior; wherein the user behavior prediction model is trained based on the following steps: obtaining multiple second historical behavior data of a sample user; converting each second historical behavior data into a behavior data text in a natural language format to obtain multiple second historical behavior data texts; training an initial model based on preset prompt words and the multiple second historical behavior data texts, and when the accuracy of the output result of the initial model meets the conditions, fine-tuning the initial model based on the anchor behavior and the balanced behavior type data to obtain the user behavior prediction model.
[0070] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the user behavior prediction method provided by the above-mentioned methods, the method comprising: inputting the first historical behavior data into a user behavior prediction model, and predicting the future behavior of the target user and the corresponding behavior type through the user behavior prediction model, wherein the behavior type includes anchor behavior and long-tail behavior, the anchor behavior includes behavior whose target type behavior data accounts for a sum of all types of behavior data greater than or equal to a preset percentage, and the long-tail behavior includes behavior other than the anchor behavior; wherein the user behavior prediction model is trained based on the following steps: obtaining multiple second historical behavior data of a sample user; converting each second historical behavior data into a behavior data text in a natural language format to obtain multiple second historical behavior data texts; training the initial model based on preset prompt words and the multiple second historical behavior data texts, and when the accuracy of the output result of the initial model meets the conditions, fine-tuning the initial model based on the anchor behavior and the balanced behavior type data to obtain the user behavior prediction model.
[0071] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.
[0072] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.
[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.
Claims
1. A user behavior prediction method, characterized in that: include: Obtain the target user's first historical behavior data; Inputting the first historical behavior data into a user behavior prediction model, and predicting the target user's future behavior and corresponding behavior type using the user behavior prediction model, wherein the behavior type includes anchor behavior and long-tail behavior, wherein the anchor behavior includes behavior for which the target type's behavior data accounts for a greater than or equal to a preset percentage of the sum of all types of behavior data, and the long-tail behavior includes behavior other than the anchor behavior; The user behavior prediction model is trained based on the following steps: Obtaining multiple second historical behavior data of sample users; Converting each of the second historical behavior data into a behavior data text in a natural language format to obtain a plurality of second historical behavior data texts; The initial model is trained based on the preset prompt words and the multiple second historical behavior data texts. When the accuracy of the output result of the initial model meets the conditions, the initial model is fine-tuned based on the anchor point behavior and the equilibrium behavior type data to obtain the user behavior prediction model.
2. The method according to claim 1, characterized in that The second historical behavior data includes a digital ID string; The step of converting each of the second historical behavior data into a behavior data text in a natural language format to obtain a plurality of second historical behavior data texts includes: According to the preset digital ID mapping table, the digital ID string of each second historical behavior data is converted into a behavior data text in natural language format to obtain multiple second historical behavior data texts, wherein the preset digital ID mapping table includes multiple digital IDs and natural language description information corresponding to each digital ID.
3. The method according to claim 2, characterized in that The training of the initial model based on the preset prompt words and the plurality of second historical behavior data texts includes: Generate prompt words based on prompt word design rules and the current model training task to obtain preset prompt words, wherein the prompt words are used by the initial model to understand the model training task; Arranging the plurality of second historical behavior data texts in chronological order according to the time corresponding to each second historical behavior data text to obtain a second historical behavior data text sequence; The second historical behavior data text sequence is input into the initial model for training, and the learning direction of the initial model is controlled by the preset prompt words.
4. The method according to claim 3, characterized in that When the accuracy of the output result of the initial model meets the conditions, the initial model is fine-tuned based on the anchor point behavior and the balanced behavior data based on the difficulty coefficient to obtain a user behavior prediction model, including: During the initial model training process, the first N second historical behaviors of the second historical behavior data text sequence are used as model input data, and the N+1 second historical behavior is used as model output data. When the accuracy of the model output data meets a threshold, the initial model training is determined to be complete, where N>1 and N is less than the data volume of the second historical behavior data text sequence; The trained initial model is fine-tuned based on anchor point behavior and balanced behavior type data to obtain a user behavior prediction model.
5. The method according to claim 4, characterized in that Fine-tuning the trained initial model based on anchor point behavior and based on balanced behavior type data to obtain a user behavior prediction model includes: Fine-tune the trained initial model using a preset anchor behavior dataset to obtain a first fine-tuned model; According to the sample selection strategy of the model prediction difficulty, a behavior prediction set pair with balanced behavior types is screened out from all behavior data sets, and the first fine-tuning model is fine-tuned based on the behavior prediction set to obtain a user behavior prediction model.
6. The method according to claim 3, characterized in that Inputting the second historical behavior data text sequence into the initial model for training includes: During the initial model training process, an autoregressive function is used to optimize and fine-tune the initial model.
7. A user behavior prediction device, characterized in that: include: An acquisition module, used to acquire the first historical behavior data of the target user; a prediction module, configured to input the first historical behavior data into a user behavior prediction model, and predict the target user's future behavior and corresponding behavior type using the user behavior prediction model, wherein the behavior type includes anchor behavior and long-tail behavior, wherein the anchor behavior includes behavior for which the target type's behavior data accounts for a greater than or equal to a preset percentage of the sum of all types of behavior data, and the long-tail behavior includes behavior other than the anchor behavior; The model training module is used to obtain multiple second historical behavior data of sample users; convert each second historical behavior data into a behavior data text in a natural language format to obtain multiple second historical behavior data texts; train the initial model based on preset prompt words and the multiple second historical behavior data texts, and when the accuracy of the output result of the initial model meets the conditions, fine-tune the initial model based on the anchor point behavior and the balanced behavior data based on the difficulty coefficient to obtain the user behavior prediction model.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the user behavior prediction method according to any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the user behavior prediction method according to any one of claims 1 to 6 is implemented.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the user behavior prediction method according to any one of claims 1 to 6 is implemented.