A human mobility prediction method and device based on a large language model cooperating with a small model plug-in

Abstract

A human flow prediction method and device based on a large language model cooperates with a small model plug-in, the method comprises the following steps: preprocessing original user check-in data, constructing a time sequence knowledge graph containing the global interaction relationship of users and places in each period, which is used to represent the check-in behavior of the user. Meanwhile, the long-term historical check-in trajectory and the recent check-in trajectory of the user are constructed; the user memory vector is constructed, and a plug-in small model composed of multiple encoders is trained. The plug-in small model is not limited to the form described in the application and can be freely replaced with other plug-in small models; the user's partial historical check-in trajectory and recent check-in trajectory are extracted, and the small model is used to generate the top K prediction labels, confidence scores, benchmark true values and other data to generate prompt words; the large language model is fine-tuned by using historical data supervision, the user's whereabouts are predicted according to the prompt without benchmark true value, and the final prediction result is obtained. The application can provide more accurate prediction results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data mining and recommendation prediction, and in particular to a human mobility prediction method and device based on a large language model cooperating with a small model plug-in. BACKGROUND

[0002] With the rapid development of mobile Internet and location-based services, users increasingly rely on location-based applications to discover and explore nearby places in their daily lives. Therefore, location-based service providers can obtain the flow trajectory of users, and human mobility refers to people moving from one place to another, usually within a geographical area, such as a city or region. Human mobility prediction is of great significance for controlling disease transmission, alleviating traffic congestion, improving travel efficiency, and commercial applications such as business site selection and place recommendation, and is an important part of smart cities.

[0003] The uniqueness of human mobility lies in its inherent regularity, randomness, and complex spatiotemporal dependence, which makes it extremely challenging to accurately predict people's whereabouts. Existing research has focused on revealing the regularity of human movement through various methods such as sequence models and graph models, i.e., leveraging the capabilities of deep learning models in spatiotemporal modeling to achieve more accurate prediction results. However, the prediction accuracy of these models still has room for improvement, and more importantly, they lack direct interpretation of their prediction results, which limits their explainability and practicality in real-world applications.

[0004] In recent years, with the rapid development of large language model technology, large models have accumulated rich common-sense knowledge covering numerous fields. This not only enables them to achieve outstanding results in various natural language processing tasks, but also exhibits impressive performance in many specialized fields under zero-shot learning conditions. However, large language models have certain theoretical limitations. First, the effective context length of these models when processing input data is limited, and they cannot process all movement trajectory data and associated data at once, which limits their ability to conduct overall statistical evaluations of user behavior on a macro level. Second, although these models have extensive general knowledge, they still struggle to fully capture and learn individual preferences in the task of predicting human mobility in specific scenarios. SUMMARY

[0005] To address the shortcomings of existing technology, a human mobility prediction method and device based on a large language model cooperating with a small model plug-in are provided. The present application improves existing solutions for real-world problems by building a large language model with embedded small models, combining the advantages of both, thereby improving the accuracy of human mobility prediction.

[0006] To achieve the above object, the first aspect of the present application relates to a human mobility prediction method based on a large language model cooperating with a small model plug-in, comprising the following steps:

[0007] S1: preprocessing the original user check-in data, constructing a time sequence knowledge graph containing the global interaction relationship between users and places in each period, for representing the check-in behavior of the user. At the same time, the long-term historical check-in trajectory and the recent check-in trajectory of the user are constructed;

[0008] S2: constructing a user memory vector and training a plug-in small model composed of multiple encoders. The plug-in small model is not limited to the form described in the present application and can be freely replaced with other plug-in small models;

[0009] S3: extracting part of the historical check-in trajectory and the recent check-in trajectory of the user, jointly generating the top K predicted labels, confidence scores and benchmark true values of each preference by the small model to generate prompt words;

[0010] S4: fine-tuning the large language model using historical data, predicting the user's whereabouts according to the prompt without benchmark true values, and obtaining the final prediction result.

[0011] Preferably, step S1 specifically comprises:

[0012] The user's check-in is processed as a check-in quadruple (u, r, l, d), where u and l are user and place entities, r is a time period relationship, and d represents a date. Specifically, the check-in quadruple reflects the check-in behavior of the user u on a specific date in a certain time period. According to the check-in time of the user, the present application divides the knowledge graph snapshot into different dates, and divides the time of a snapshot into multiple equal time periods, thereby obtaining the time period relationship of the user's check-in. In addition, the previous check-in of each check-in of the user is saved, and the global human flow relationship between places is constructed based on this. In addition, the present application needs to construct a historical check-in trajectory with a length of M and a recent check-in trajectory with a length of N for each user.

[0013] As a preferred, step S2 specifically comprises:

[0014] S2.1: constructing the historical check-in frequency of all places of the user in different time periods into a user memory vector. Specifically, for each user check-in record, the present application traverses all historical check-in data before its timestamp t to calculate the check-in memory vector of the user for each place As shown in formula (1).

[0015]

[0016] Wherein, t i and t j represent the check-in record of the user u and ti j timestamp, r represents the check-in time period relationship of the user, and d represents the time information of the check-in quadruple, that is, the date number. Next, we convert to

[0017]

[0018] S2.2: A learning strategy based on a replication mechanism is adopted to model the user's personal preferences from two aspects. One is the user's habit, and the other is the user's potential exploration demand for checking into a new place. For the user's habit preference, existing research observes that the user's daily life has certain rules, which means that researchers can capture the user's habit preference through the time information of the current query context and the user's historical check-in data. Therefore, the user entity embedding and the time relationship embedding are connected as query features, and then a latent habit preference vector is generated using it, as follows:

[0019]

[0020] where tanh is an activation function, and are trainable parameters, L is a place embedding vector, represents an embedding splicing operation. It is worth noting that the is added to the habit preference vector, which divides the candidate space into two parts, so that the habit preference vector pays more attention to the places that have been checked in before.

[0021] However, due to factors such as daily life needs, curiosity about new things, and the influence of popular events, users may also be interested in places they have never checked in before. Therefore, in contrast to the user's habit preference, the invention also needs to capture the user's novelty preference. According to the same logic as modeling the user's habit preference, the user's novelty preference vector is obtained as follows:

[0022]

[0023] In contrast to the habit preference, the invention adds the negative number of to the novelty preference vector, so that the novelty preference vector pays more attention to places that the user has never checked in before.

[0024] S2.3: For each check-in quadruple (u, r, l, d), the invention saves the previous check-in record, that is, (u, r i ,l i , d i , r j , l j , d j). To obtain the time information between sequential check-ins, the application obtains the difference between the dates of two check-ins and the unit date vector d u The time difference embedding is generated. Finally, the application combines the embedding of sequential check-ins to obtain the place flow transfer feature vector:

[0025] Ad = (d i -d j ) x d u (5)

[0026]

[0027] S2.4: The obtained various preference vectors are activated by a Soft-max activation function to obtain the stage prediction result, as shown in the following formula:

[0028] P = σ (V) (7)

[0029] As a preferred, in step S3, to construct the prompt, the application selects M records from the historical check-ins to form a historical check-in sequence, and selects N records from the recent check-ins to form a recent check-in sequence. In addition, the application uses a large language model to generate a limited length user profile summary based on the user's historical data. The profile summarizes the user's personal information, including gender, age, education level, economic status, and interest preferences, and other key information.

[0030] Then, the application performs a top-k precision analysis on each preference prediction in the plug-in model, and we find that when the plug-in model predicts the top 30 most likely places, it can achieve an accuracy of more than 80%, and when more prediction items are selected, the relative accuracy improvement is not obvious. Therefore, after comprehensive evaluation, the application selects the top 30 most likely prediction results as candidates to avoid generating too long prompt contexts.

[0031] Finally, these data are input as prompt words for predicting human flow.

[0032] As a preferred, step S4 specifically includes:

[0033] S4.1: Fine-tuning a large language model can have high costs, so the application applies a parameter-efficient fine-tuning (PEFT) technique during the fine-tuning phase. The application applies a low-rank adaptation algorithm (Low-Rank Adaptation, LoRA) to freeze the dense layers in the large language model and update the weight values with rank decomposition matrices. This allows only a small number of parameters to receive gradient updates during training.

[0034] S4.2: In order to reduce the excessive memory overhead, the application adopts a double quantization method, that is, Normal Float4 bit quantization is used for storage, and Brain Floating 16 bit quantization is used for forward and backward propagation.

[0035] S4.3: Since a large amount of check-in trajectory data needs to be given in the prompt word when human mobility prediction is performed, a longer prompt word context length is required. Therefore, the application applies the FlashAttention-2 technology to enable Transformers to accept longer context.

[0036] S4.4: After obtaining the supervised fine-tuned large language model, the application uses the prompt word without label data to make prediction to obtain the final prediction result.

[0037] The second aspect of the application relates to a human mobility prediction device based on a large language model cooperating with a small model plug-in, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the human mobility prediction method based on the large language model cooperating with the small model plug-in of the application.

[0038] The third aspect of the application relates to a computer readable storage medium having a program stored thereon, wherein the program is executed by a processor to implement the human mobility prediction method based on the large language model cooperating with the small model plug-in of the application.

[0039] The application analyzes the check-in data of the user to construct a user track time sequence knowledge graph, aims to realize a lightweight and efficient traditional model, extracts the historical trajectory and recent trajectory of the user, and combines the data of the plug-in traditional model as the input prompt word of the large language model, the pre-trained large language model has a rich common sense knowledge base, and deeply understands the implicit knowledge in the context information which is difficult to mine by the traditional method, in addition, the large language model can explain the prediction result, which solves the deficiency of the traditional method in explaining the prediction result, and the plug-in model effectively solves the limitation of the large language model in processing long context and mining non-common sense preferences. The application can provide more accurate prediction results.

[0040] Compared with the prior art, the application mainly has the following beneficial effects:

[0041] 1. The plug-in small model constructed based on the proposed user trace timing knowledge graph is a new timing knowledge graph reasoning model, which can be used for individual human mobility prediction tasks. It uses a context-aware replication mechanism to identify user habits and novelty preference, and learns from global user data through a simple model structure instead of complex graph convolution, effectively reducing the problem of data sparsity and solving the user cold start problem.

[0042] 2. The present application proposes a technical architecture aiming to coordinate pre-training large language models with plug-in models to realize the prediction of human mobility. The architecture utilizes the extensive common sense knowledge contained in large language models to provide many beneficial thoughts for human mobility prediction. The present application integrates small deep learning models constructed using traditional methods as plug-ins, and uses the prediction results generated by these small models as input prompts to activate the common sense knowledge of large language models. In addition, the present application can generate corresponding prediction reasons using the natural language processing capabilities of large language models while predicting human mobility. In summary, the present application not only effectively makes up for the limitations of traditional methods, but also significantly improves the accuracy of human mobility prediction. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 Figure 1 is a model structure diagram of the human mobility prediction method based on large language model coordination with small model plug-in of the present application.

[0044] Figure 2 Figure 2 is a flowchart of the human mobility prediction method based on large language model coordination with small model plug-in of the present application.

[0045] Figure 3 Figure 3 is a device schematic diagram of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0047] It should be noted that the features in the following embodiments and implementation manners can be combined with each other without conflict.

[0048] As shown in Figure 1 and Figure 2 , the present application proposes a human mobility prediction method based on large language model coordination with small model plug-in, which specifically includes the following steps:

[0049] S1: preprocessing the original user check-in data, constructing a time sequence knowledge graph containing the global interaction relationship between users and places in each period, for representing the check-in behavior of the user. Meanwhile, constructing the long-term historical check-in trajectory and the recent check-in trajectory of the user;

[0050] S2: constructing a user memory vector and training a plug-in small model composed of multiple encoders. The plug-in small model is not limited to the form described in the present application and can be freely replaced with other plug-in small models;

[0051] S3: extracting part of the historical check-in trajectory and the recent check-in trajectory of the user, and jointly generating the top K predicted labels, confidence scores and benchmark true values of each preference by the small model to generate prompt words;

[0052] S4: using the historical data to supervise the fine-tuning of the large language model, predicting the user's whereabouts according to the prompt without benchmark true values, and obtaining the final prediction result.

[0053] The above-mentioned human mobility prediction method based on a large language model cooperating with a small model plug-in, S1 specifically includes the following steps:

[0054] The check-in of the user is processed into a check-in quadruple (u, r, l, d), where u and l are user and place entities, r is a time period relationship, and d represents a date. Specifically, the check-in quadruple reflects the check-in behavior of the user u in a specific time period on a certain date. According to the check-in time of the user, the present application divides the knowledge graph into snapshots of different dates, and divides each snapshot into multiple equal time periods according to the time, thereby obtaining the time period relationship of the user's check-in. In addition, the previous check-in of each check-in of the user is saved, and the global human flow relationship between places is constructed based on this. In addition, the present application needs to construct a historical check-in trajectory with a length of M and a recent check-in trajectory with a length of N for each user.

[0055] As a preferred, the step S2 specifically includes:

[0056] S2.1: constructing the historical check-in frequency of all places of the user in different time periods into a user memory vector. Specifically, for each user check-in record, the present application traverses all historical check-in data before the time stamp t of the user to calculate the check-in memory vector of the user to each place As shown in formula (1).

[0057]

[0058] Wherein, t i and t j represent the check-in record of the user u and t i t jtimestamp, r represents the user's check-in time period relationship, T represents the time information of the check-in quadruple, that is, the date number. Next, we convert it to

[0059]

[0060] S2.2: A learning strategy based on a replication mechanism is adopted to model the user's personal preferences from two aspects. One is the user's habits, and the other is the user's potential exploration demand for new places. For the user's habit preference, existing research observes that the user's daily life has certain rules, which means that researchers can capture the user's habit preference through the time information of the current query context and the user's historical check-in data. Therefore, the present application connects the user entity embedding and the time relationship embedding as query features, and then uses it to generate a latent habit preference vector, as follows:

[0061]

[0062] where tanh is an activation function, and are trainable parameters, L is a place embedding vector, represents an embedding concatenation operation. It is worth noting that the present application adds to the habit preference vector, which divides the candidate space into two parts, so that the habit preference vector pays more attention to the places that have been checked in before.

[0063] However, due to factors such as daily life needs, curiosity about new things, and the influence of popular events, users may also be interested in places they have never checked in before. Therefore, in contrast to the user's habit preference, the present application also needs to capture the user's novelty preference. According to the same logic as modeling the user's habit preference, the user's novelty preference vector is obtained as follows:

[0064]

[0065] In contrast to the habit preference, the present application adds the negative number of to the novelty preference vector, so that the novelty preference vector pays more attention to places that the user has never checked in before.

[0066] S2.3: For each check-in quadruple (u, r, l, d), the present application saves the previous check-in record, that is, (u, r i ,l i , d i , r j , l j , d j). To obtain the time information between sequential check-ins, the application obtains the difference between the dates of two check-ins and the unit date vector d u The time difference embedding is generated. Finally, the application combines the embedding of sequential check-ins to obtain the place flow transfer feature vector:

[0067] Ad = (d i -d j ) x d u (5)

[0068]

[0069] S2.4: The obtained various preference vectors are activated by a Soft-max activation function to obtain a stage prediction result, as shown in the following formula:

[0070] P = σ (V) (7)

[0071] As a preferred, in step S3, to construct the prompt word, the application selects M records from the historical check-ins to form a historical check-in sequence, and selects N records from the recent check-ins to form a recent check-in sequence. In addition, the application uses a large language model to generate a limited length user profile summary based on the user's historical data. The profile summarizes the user's personal information, including gender, age, education level, economic status, and interest in key information.

[0072] Then, the application performs a top-k precision analysis on each preference prediction in the plug-in model, and we find that when the plug-in model predicts the top 30 most likely places, it can achieve an accuracy of more than 80%, and when more prediction items are selected, the relative accuracy does not improve significantly. Therefore, after comprehensive evaluation, the application selects the top 30 most likely prediction results as candidates to avoid generating too long prompt word context.

[0073] Finally, these data are input as prompt words for predicting human flow, as shown in the following partial prompt words:

[0074] <question>Your task is to conduct a comprehensive analysis of User{UID}'s preferences and then consider which POI id the user will visit attime{time}?

[0075] Given the user’s history check-ins sequence:{history_sequence}(format:[(PID,Category,Time),...]).

[0076] Given the user’s recent check-ins sequence:{recent_sequence}(format:[(PID,Category,Time),...]).

[0077] Given the plugin model's predictions of the user's preferences attime{time}:{plugin_predictions}(format:{"Specific Preference”:[(PID,Category,Distance,Confidence),...],...})

[0078] Now let me explain the elements in the format."PID”refers to theunique ID of the POI,"Distance”represents the distance(in kilometers)betweenthe user's last visited address and the current POI,and"Category”representsthe semantic information of the POI."Confidence”represents the confidencelevel of the plugin model's predicted results.

[0079] <answer>At time{time},user{UID}will visit POI{gt_PID}which is a / an{gt_Category}.

[0080] As preferred, the step S4 specifically comprises:

[0081] S4.1: Fine-tuning large language models can have high costs, so the application applies parameter-efficient fine-tuning (PEFT) technology in the fine-tuning stage. The application applies the low-rank adapter algorithm (Low-Rank Adaptation, LoRA) to freeze the dense layers in the large language model and update the weight values with rank decomposition matrices. During training, only a small number of parameters will receive gradient updates.

[0082] S4.2: To reduce excessive memory overhead, the application uses a double quantization method, that is, Normal Float4 bit quantization is used for storage, and Brain Floating 16 bit quantization is used for forward and backward propagation.

[0083] S4.3: Since a large amount of check-in trajectory data needs to be given in the prompt word when human mobility prediction is performed, a longer prompt word context length is required. Therefore, the application applies FlashAttention-2 technology to enable Transformers to accept longer contexts.

[0084] S4.4: After obtaining the supervised fine-tuned large language model, the application uses the prompt word without label data to make predictions to obtain the final prediction result.

[0085] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

[0086] The above describes the present application and its embodiments, which are not restrictive, and the embodiments shown in the drawings are only one of the embodiments of the present application, and the actual structure is not limited thereto. In general, if a person skilled in the art is inspired by it, without departing from the purpose of the present application, without creative design, similar structure and embodiments of the technical solution can be designed and belong to the protection scope of the present application.< / answer> < / question>

Claims

1. A human mobility prediction method based on a large language model cooperated with a small model plug-in, specifically comprising the following steps: S1: preprocessing the original user check-in data, constructing a time sequence knowledge graph containing the global interaction relationship between users and places in each period, for representing the check-in behavior of the user; at the same time, constructing the long-term historical check-in trajectory and the recent check-in trajectory of the user; S2: constructing a user memory vector and training a plug-in small model composed of multiple encoders; the plug-in small model is not limited to the form described in the present application and can be freely replaced with other plug-in small models; specifically comprising: S2.1: Construct the history check-in frequency of all locations of the user in different time periods as a user memory vector; specifically, for each user check-in record, traverse all the history check-in data before its timestamp t to calculate the user's check-in memory vector for each location As shown in formula (1); Among them, t i and t j Represents the check-in record of user u and t i >t j timestamp, r represents the user's check-in time period, and d represents the time information of the check-in quadruple, that is, the date number; Next, according to the following formula, Convert to S2.2: adopting a learning strategy based on a replication mechanism to model the personal preferences of the user from two aspects; one is the habit of the user and the other is the potential exploration demand of the user when checking into a new place; the habit preference of the user is captured through the time information of the current query context and the historical check-in data of the user; therefore, the user entity embedding and the time relationship embedding are connected as the query features, and then the latent habit preference vector is generated using it, as follows: where tanh is the activation function, and are trainable parameters, L is the location embedding vector, denotes the embedding concatenation operation; it is worth noting that adding to the habit preference vector splits the candidate space in two, making the habit preference vector more focused on locations that have been checked-in before; capture the novelty preference of the user; according to the same logic as modeling the habit preference of the user, the user's novelty preference vector is obtained, as follows: Adding a negative of the number of times the user has checked in at a location to the try- new preference vector makes the try-new preference vector pay more attention to locations that the user has never checked in at before. the try-new preference vector makes the try-new preference vector pay more attention to locations that the user has never checked in at before. S2.3: For each check-in quadruple (u, r, l, d), the previous check-in record, i.e., (u, r i ,l i ,d i ,r j ,l j ,d j ) is saved; to obtain the time information between sequential check-ins, the time difference embedding is generated according to the date difference of two check-ins and the unit date vector d u ; finally, the place flow transfer feature vector is obtained in combination with the embedding of sequential check-ins: Ad = (d i -d j ) x d u (5) S2.4: obtaining the stage prediction result by passing the obtained various preference vectors through a Soft-max activation function, as shown in the following formula: P=σ(V) (7) S3: extracting part of the historical check-in trajectory and the recent check-in trajectory of the user, generating the top K prediction labels, confidence scores and benchmark true value data of each preference, and generating the prompt word; S4: using the historical data to supervise the fine-tuning of the large language model, predicting the whereabouts of the user according to the prompt without benchmark true value, and obtaining the final prediction result.

2. The human mobility prediction method based on a large language model cooperating with a small model plug-in according to claim 1, wherein, Step S1 specifically comprises: processing the check-in of the user into a check-in quadruple (u, r, l, d), wherein u and l are user and place entities, r is a time period relationship, and d represents a date; specifically, the check-in quadruple reflects the check-in behavior of the user u on a specific time period of a date; according to the check-in time of the user, the knowledge graph snapshot of different dates is divided, and according to the time of a snapshot, multiple equal time periods are divided, thereby obtaining the time period relationship of the user check-in; in addition, the previous check-in of each check-in of the user is saved, and the global human flow relationship between places is constructed; in addition, a historical check-in trajectory with a length of M and a recent check-in trajectory with a length of N are constructed for each user.

3. The human mobility prediction method based on a large language model cooperating with a small model plug-in according to claim 1, wherein, In step S3, to construct the prompt word, M records are selected from the historical check-in to form a historical check-in sequence, and N records are selected from the recent check-in to form a recent check-in sequence; in addition, a user profile summary with a limited length is generated based on the historical data of the user by using the large language model; the profile summarizes the personal information of the user, including gender, age, education level, economic status and interest key information; Then, the accuracy of each preference prediction in the plug-in model is analyzed, and the top k prediction results are selected to avoid generating too long prompt word context; Finally, these data are input as prompt words for predicting human mobility.

4. The human mobility prediction method based on a large language model cooperating with a small model plug-in according to claim 1, wherein, The step S4 specifically comprises: S4.1: applying a parameter-efficient fine-tuning PEFT technique in the fine-tuning stage; applying a low-rank adapter algorithm LoRA, freezing the dense layers in the large language model, and updating the weight values with a rank decomposition matrix; so that only a small number of parameters will receive gradient updates during training; S4.2: a double quantization method is adopted, Normal Float 4-bit quantization is used for storage, and Brain Floating 16-bit quantization is used for forward and backward propagation; S4.3: applying FlashAttention-2 technology enables Transformers to accept longer context; S4.4: after obtaining the supervised fine-tuned large language model, the prompt word without label data is used for prediction to obtain the final prediction result.

5. A human mobility prediction device based on a large language model cooperated with a small model plug-in, characterized in that, The device comprises a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to implement the human flow prediction method based on the large language model cooperated with the small model plug-in.

6. A computer-readable storage medium, characterized in that, A program is stored thereon, and the program is executed by a processor to implement the human flow prediction method based on the large language model cooperated with the small model plug-in.

Images