A method for inferring freight stop hotspot sequences based on BERT language model
Through the inference method of freight stay hotspot sequence based on the BERT language model, using contrast learning and data enhancement technology, the problem of low stop preference capture and low trajectory prediction accuracy in the existing technology is solved, and a more accurate stay hotspot sequence is achieved Inference and adaptability improve.
Patent Information
- Application Number
- CN202410402502.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-03
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-04-03
AI Technical Summary
The prior art is difficult to accurately capture the stay preferences and infer the stay hotspot sequence of truck drivers, especially when the prediction accuracy is not high under flow directions with a small number of trajectories, and spatial heterogeneity and similarity are not fully considered.
The freight stay hotspot sequence inference method based on the BERT language model is adopted. By extracting multiple feature attributes of the stay hotspot, coding and pre-training, combining comparative learning tasks and data enhancement, the model is fine-tuned to improve prediction accuracy.
It realizes a more accurate capture of the stop preferences of truck drivers and infers the stop hotspot sequence, improves the prediction accuracy of flow directions with a small number of trajectories, and adapts to spatial heterogeneity and similarity of different flow directions.
Smart Images

Figure BDA0004776061590000051 
Figure BDA0004776061590000052 
Figure BDA0004776061590000053
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of trajectory mining, and in particular relates to a method for inferring a freight stop hotspot sequence based on a BERT language model. Background Art
[0002] In the bulk freight scenario, route recommendation plays a vital role. A good route recommendation scheme can significantly improve the satisfaction of truck drivers, thereby improving logistics efficiency and reducing operating costs. Unlike the driving routes of ordinary vehicles in urban road networks, the driving routes of trucks in bulk logistics scenarios are characterized by long mileage, long time and cross-provincial and municipal road networks. A long-distance transportation task takes dozens of hours or even days. Therefore, in order to meet the needs of rest and meals during transportation, truck drivers will choose suitable places to stop. The purpose of stop hotspot sequence inference is to infer the sequence of stop hotspots visited by the driver during the transportation task based on the driver's historical trajectory data and the current driver's query request. Accurate stop hotspot sequence inference can better assist route planning, prevent driver fatigue driving and meet the driver's various stop needs, thereby better improving the driver's satisfaction with the route. In addition, it also helps to improve the accuracy of transportation time prediction and facilitate the arrangement of subsequent transportation plans. However, there is currently a lack of a stop hotspot sequence inference method that meets the stop preferences of truck drivers.
[0003] It is not easy to accurately capture the driver's stop preference from historical data and infer the sequence of stop hotspots. First of all, the driver's choice of the current stop hotspot will be affected by a variety of complex factors such as the starting and ending points, the previous stop hotspots, the departure period, and the working period restrictions at the transportation end. How to fully model the influence of these complex factors and infer the current stop hotspot is the first challenge faced by existing technologies. In addition, in the bulk logistics scenario, there is a skewed distribution of trajectory data for different flows, where the flow direction represents the direction from the starting point to a certain end point. The number of historical trajectories of some popular flows is much larger than that of unpopular flows, resulting in unbalanced trajectory data for different flows. Existing methods tend to infer the stop hotspot sequence of flows with a large number of trajectories, while the inference accuracy of flows with a small number of trajectories is not high. Therefore, how to improve the prediction accuracy of flows with a sparse number of trajectories is the second challenge faced by existing technologies.
[0004] The task of inferring hotspot sequences is similar to the task of itinerary recommendation in the field of POI recommendation. Itinerary recommendation aims to recommend a sequence of points of interest (POI) that meet the user's preferences based on the user's query request. Existing itinerary recommendation methods can be divided into two categories: traditional machine learning-based methods and deep learning-based methods. Traditional machine learning-based methods mostly use simple statistical methods to model user behavior patterns [1-2]. For example, some methods use Markov chains to learn the transition probability between POIs from historical trajectory data [1]. Other methods regard the itinerary recommendation problem as a variant of the directional problem [2]. They use the popularity of POIs and the user's visit time at POIs to score POIs and recommend routes. However, the above methods rely on the objective function and constraint function set by humans a priori. Due to the diversity of user needs and the sparsity of historical trajectory data, the above methods cannot well describe the user's true travel intentions.
[0005] In recent years, Seq2Seq models have been widely used in fields such as natural language processing, and have achieved great success in tasks such as machine translation. At the same time, since the input of itinerary recommendation can be regarded as a sequence containing only the starting and ending points, and the output can be regarded as a POI sequence, many scholars have modeled itinerary recommendation as a Seq2Seq problem [3-5]. The methods used include inference methods based on RNN and GAN [3], methods based on Transformer and multi-task learning [4], and methods based on contrastive learning [5]. In addition, some scholars have regarded the itinerary recommendation task as a cloze task and proposed a self-supervised itinerary recommendation framework based on BERT [6], which makes full use of the attention mechanism for itinerary recommendation. However, the above methods do not take into account the spatial heterogeneity and similarity of trajectory data in different directions. Spatial heterogeneity means that the distribution of stop hotspots in some directions is different in geographic space, which will affect the truck drivers' choice of stop hotspots; similarity means that the distribution of stop hotspots in some directions is similar in geographic space, so the stop hotspot sequences of drivers in these directions will be closer. At the same time, the above methods all rely on the ID features of POIs. However, the IDs of hotspots in different directions are different. For example, in a certain direction, the route traveled by driver dri1 is 1-2-3 (the numbers represent the IDs of hotspots). The previous method can capture the sequential dependency between 1-2-3, but such information cannot be transferred to other directions. Therefore, it cannot solve the problem of sparse trajectory data and cold start. Summary of the invention
[0006] In order to address the deficiencies in the prior art, the purpose of the present invention is to provide a freight stop hotspot sequence inference method based on the BERT language model, which can capture the truck driver's stop preference by using the truck driver's historical trajectory data, so as to better serve the transportation route planning and transportation time prediction.
[0007] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:
[0008] A method for inferring freight stop hotspot sequences based on a BERT language model includes the following steps:
[0009] S1: Extraction of hotspot attributes: Extract multiple characteristic attributes of hotspots from the hotspot sequence, including hotspot function type, departure time, stay duration and geographic location;
[0010] S2: Stay hotspot sequence division stage: re-dividing the historical stay hotspot sequence to align the inputs during training and inference;
[0011] S3: Stay hotspot sequence embedding stage: Based on the stay hotspot sequence obtained in S2, four learnable encoding layers are used to encode the stay hotspot sequence as the input of the subsequent pre-training model;
[0012] S4: Pre-training model learning stage: Based on the BERT language model, two contrastive learning proxy tasks are introduced to improve the BERT model, and the improved BERT language model is used to pre-train the stay hotspot encoding obtained based on S3 to obtain the stay hotspot pre-training representation;
[0013] S5: fine-tuning of the representation of the stay hotspot: the historical data of the current flow direction is used to fine-tune the pre-trained representation of the stay hotspot obtained based on S4 to obtain the final stay hotspot sequence prediction model;
[0014] S6: Stop hotspot sequence prediction stage: Based on the final stop hotspot sequence prediction model obtained in S5, the next stop hotspot that the driver wants to stay at is predicted in sequence according to the query request given by the driver until the end point is predicted.
[0015] To optimize the above technical solutions, the specific measures taken also include:
[0016] In the above step S1,
[0017] In order to fully consider the influence of the stop hotspot function type, time dimension (including departure time, stop duration, etc.) and geographic space dimension (including geographic space location, etc.) on the driver's choice of stop hotspot in transportation tasks, the present invention extracts four attributes from the stop hotspot sequence, including: 1) the stop hotspot function type (expressed as SH type): The present invention divides the functional types of the stopover hotspots into five categories (including origin, destination, rest area, restaurant and gas station); 2) departure time (expressed as SH departure ): The present invention divides a day [00:00, 24:00) into 24 intervals, namely [00:00, 01:00), [01:00, 02:00), ..., [23:00, 24:00); 3) duration of stay (expressed as SH duration ): The present invention sets three stay time intervals, namely [0 minutes, 40 minutes), [40 minutes, 60 minutes) and [60 minutes, ∞); 4) geographic spatial location (expressed as SH space ): The present invention divides the map into grids, and maps the latitude and longitude coordinates of the hotspot to the grid ID of the location of the hotspot;
[0018] In a specific implementation, the selected map is a map of Shandong Province.
[0019] In the above step S2,
[0020] Since all the stop locations in each transportation task can be known in advance during training, but only the starting point and end point information provided by the driver is available during reasoning, in order to align the inputs during training and reasoning, the present invention re-combines the stop hotspot sequence obtained based on S1, and virtually copies the end point information after the starting point of the stop hotspot sequence used for training, so that the stop hotspot sequence used for training and the sequence input format during reasoning are consistent. At the same time, in order to expand the amount of data so that the subsequent model can be better trained and converged, the present invention performs data enhancement on the combined stop hotspot sequence used for training. Specifically, for a stop hotspot sequence of any length n, the present invention adopts a dynamic interception strategy for data enhancement, that is, each time the first i stop points are extracted as components of the enhanced sequence, where the stop points include the starting and ending points and the stop hotspots along the way, where i increases from a minimum of 3 to a maximum of n, so as to generate a diverse set of training samples; that is, the enhanced sequence includes a sequence from the starting point to the third stop point, a sequence from the starting point to the fourth stop point, ..., a sequence from the starting point to the nth stop point, etc.
[0021] In the above step S3,
[0022] In view of the rapid development of the field of natural language processing, various language models have been proven to have very powerful representation capabilities. The task of inferring the sequence of stop hot spots is similar to some tasks in the field of natural language processing. For example, in the task of sentence translation, each word in the sentence will be affected by the words in the previous and next contexts. This is similar to the fact that the driver's choice of the current stop hot spot will be affected by the starting and ending points and the previous stop hot spots considered in the present invention. Therefore, the present invention considers using a language model to model the stop hot spot sequence recommendation task. Based on the stop hot spot attribute characteristics obtained in step S1, the present invention constructs each stop hot spot into one or more key-value attribute pair sequences, wherein each key or value represents a word element, which respectively represents the key-value attributes of the stop hot spot function type, departure time, stop duration and geographic spatial location in the stop hot spot feature attributes. In addition, in order to consider the personalized differences of drivers, the present invention also adds the 'Driver ID' word element to the stop hot spot attribute characteristics to distinguish different drivers. Finally, the present invention also adds the 'CLS' word element at the beginning of the stop hot spot sequence to represent the entire stop hot spot sequence. After that, the present invention uses four learnable encoding layers to encode the entire stop hot spot sequence, namely, the stop hot spot position encoding layer E ... sh , word type encoding layer E tt , word unit position encoding layer E tp and word encoding layer E t . The stop hotspot position encoding layer is used to locate the positions of the word units that make up different stop hotspots, which helps the model learn the sequential dependencies between stop hotspots; the word unit type encoding layer is used to distinguish different types of word units, which helps to learn the influence of different types of features on the next stop hotspot; the word unit position encoding layer is used to locate the positions of different word units, which is similar to the stop hotspot position encoding layer, in order to better capture the sequential dependencies at the word unit level; the word unit encoding layer is used to encode specific word units. Finally, the embedding formula corresponding to the stop hotspot sequence is obtained as follows:
[0023] E emb =E sh +E tt +E tp +E t .
[0024] In the above step S4:
[0025] Considering that the BERT language model has been widely used in the field of natural language processing and has achieved good results, the present invention chooses to use BERT to further characterize the dwell hotspot sequence embedding. BERT uses the Transformer Encoder architecture, and the specific calculation formula of BERT is as follows:
[0026] [x [CLS] ,x [DRIVER],x [emb1] ,...,x [embn] ]=BERT([E [CLS] ,E [DRIVER] ,E [emb1] ,...,E [embn] ]),
[0027] The vector x represents the output of the BERT model. [CLS] ∈R h Used to represent the entire sequence of hotspots, x [DRIVER] represents the representation of the driver word output by the BERT model, x [embn] represents the representation of the nth attribute word output by the BERT model, E [CLS] The embedding vector representing the word ‘CLS’, E [DRIVER] Represents the embedding vector of the driver word, E [embn] represents the embedding vector of the nth attribute word, and h represents the dimension of the embedding.
[0028] and / or,
[0029] This paper follows the design of the masked language task in BERT. For a given stop hotspot sequence input, 15% of the tokens are randomly selected for masking, that is, replaced with the '[MASK]' token. The loss calculated by the masked language model is as follows:
[0030] p = Softmax(GELU(W1x i +b1)W2+b2),
[0031]
[0032] Where W1∈R h×h is a learnable parameter, b1∈R h and b2∈R |V| is the bias term, W2∈R h×|V| is the embedding matrix of the dwell hotspot, |V| is the number of dwell hotspots, Softmax() is the softmax activation function, GELU() is the GELU activation function, x i is the representation vector of the i-th word, x i represents the representation vector of each word after BERT, p is the result after softmax function processing, representing the probability of each word being selected, y i is a binary variable. If the word at position i is predicted correctly, then y i =1, otherwise y i =0, p iIt is the predicted probability of the i-th word calculated by the softmax function, which indicates the probability that the model believes that the word at position i belongs to its predicted category.
[0033] In addition, two contrastive learning proxy tasks are introduced in step S4 to improve the BERT model:
[0034] For the contrastive learning task between the stop hotspot sequence and the stop hotspot, the purpose of this contrastive learning task is to further enhance the sequential dependency between the visited stop hotspot sequence and the next stop hotspot. During training, each sample is recorded as (x seq ,x sh ), where x seq represents the sequence representation of the previously visited hotspots, x sh Represents the next hotspot to be visited. For a training sample in a batch {(x seq1 ,x sh1 ),...,(x seqb ,x shb )}, the present invention uses the previously visited stop hotspot sequence representation and the real next stop hotspot representation corresponding to the previously visited stop hotspot sequence representation as a positive sample, and the next stop hotspot representation combination to be visited in other samples as a negative sample. The loss function calculation formula of the stop hotspot sequence and stop hotspot comparison learning task is as follows:
[0035]
[0036]
[0037] Where cos represents the cosine similarity between two vectors, B represents the batch, and τ represents the temperature coefficient.
[0038] For the comparative learning task between stay hotspot sequences, the stay hotspot sequences with similar stay patterns and stay hotspot sequences with large differences in stay patterns are further distinguished from the level of stay hotspot sequences. Specifically, the present invention designs three data enhancement methods: 1) Stay hotspot position offset: randomly select a stay hotspot from the stay hotspot sequence, and randomly select a stay hotspot of the same functional type near its position to replace it; 2) Stay hotspot random masking: randomly select a stay hotspot from the stay hotspot sequence, and use the '[MASK]' word to replace all the words representing the selected stay hotspot; 3) Word random masking: use the '[MASK]' word to randomly mask out the words in the stay hotspot sequence. During training, given a stay hotspot sequence representation A new representation of the dwell hotspot sequence is obtained by any one or more of the above data enhancement methods. As a positive sample, the stay hotspot sequence representation obtained by enhancing other stay hotspot sequences in the same batch is used as a negative sample. The loss function calculation formula of the comparison learning task between the stay hotspot sequences is as follows:
[0039]
[0040] Contrastive learning within a batch, where cos represents the cosine similarity between two vectors, B represents the batch, and τ represents the temperature coefficient.
[0041] Finally, the present invention adopts a multi-task learning method to train the model, that is, the loss functions of the masked language model task, the comparison learning task between the stop hotspot sequence and the stop hotspot sequence, and the comparison learning task between the stop hotspot sequences are accumulated as the overall loss function of the pre-trained model. The overall loss function calculation formula is as follows:
[0042] L all =L mask +αL seq-sh +βL seq-seq ,
[0043] Where α and β are hyperparameters.
[0044] The above step S5 specifically includes:
[0045] Taking into account the spatial heterogeneity of the stay patterns of different flow directions, the present invention only uses the data of a single flow direction to fine-tune the pre-trained model obtained based on step S4. In order to enable the model to better capture the higher-level stay patterns between different flow directions, rather than the sequential dependencies between specific stay hotspot IDs, the present invention does not extract the ID features of the stay hotspots in the pre-training stage. However, the fine-tuning stage is only trained on the data set of a single flow direction. At this time, adding the ID features of the stay hotspot can better improve the accuracy of inference; in addition, the contrast learning used in pre-training is performed within a batch, and all negative samples are not taken into account, which inevitably leads to some errors. To this end, in the fine-tuning stage, the present invention first obtains all the stay hotspots of the current fine-tuning flow direction, and inputs them into the trained pre-training model to obtain the representation x of each stay hotspot. sh , and then the embedding layer is used to obtain the representation k of each hotspot ID feature sh , the formula of the fine-tuning model loss function is as follows:
[0046]
[0047] Where M represents the number of hot spots in the current flow, cos represents the cosine similarity between two vectors, B represents the batch, and τ represents the temperature coefficient.
[0048] The above step S6 specifically includes:
[0049] In the hotspot sequence prediction stage, the present invention firstly predicts the sequence of all flow direction models MS, the hotspot set SS and the upper limit of the iteration threshold th based on the fine-tuned up , the driver inputs a query request q, wherein the query request refers to the request issued by the truck driver when performing each transportation task, which is expressed as q=(Ori, Dest, DestCity, to, td, Dri), wherein Ori and Dest represent the starting point and the end point respectively, DestCity represents the end point city, to and td represent the departure time and the expected arrival time respectively, and Dri represents the truck driver. The present invention extracts the original stop hotspot sequence containing only the starting point and the end point according to the Ori and Dest fields in the query request q, and determines the flow direction of the current query according to the DestCity field in the query request q, and finds the fine-tuned model and stop hotspot set corresponding to the flow direction. Then, the original stop hotspot sequence is sent to the fine-tuned model of the corresponding flow direction, and the score of each candidate stop hotspot is calculated using cosine similarity. The stop hotspot with the highest score is inserted into the original stop hotspot sequence to form a new stop hotspot sequence, and then the above steps are performed in sequence until the predicted stop position is the end point or the algorithm reaches the iteration limit.
[0050] The beneficial effects of the present invention include:
[0051] In view of the challenge that the driver's choice of the current stop hotspot is affected by multiple complex factors, the present invention regards the stop hotspot sequence inference problem as a language model task and uses the improved BERT model to fully capture the influence of complex factors on the inference of the current stop hotspot.
[0052] To address the problem of low inference accuracy due to sparse data of some flow trajectories, the present invention uses pre-training and fine-tuning to transfer knowledge of driver stop patterns to improve the accuracy of inference. Experiments are conducted on real experimental data to verify the effectiveness of the method proposed in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0054] Figure 1 It is a framework diagram of a stop hotspot sequence inference method based on the BERT language model.
[0055] Figure 2It is a stay hotspot attribute extraction diagram of the method of the present invention in an embodiment of the present invention.
[0056] Figure 3 It is a stop hotspot sequence division diagram of the method of the present invention in an embodiment of the present invention.
[0057] Figure 4 It is a dwell hotspot sequence embedding diagram of the method of the present invention in an embodiment of the present invention.
[0058] Figure 5 Schematic diagram of the proxy task of the method of the present invention in an embodiment of the present invention.
[0059] Figure 6 1 is a schematic diagram of the stop hotspot sequence prediction process of the method of the present invention in an embodiment of the present invention. DETAILED DESCRIPTION
[0060] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.
[0061] The present invention provides a method for inferring a sequence of freight stop hotspots based on a BERT language model. The method first extracts the functional type, departure period, stop duration, and geographic location of each stop hotspot, and imitates some tasks in natural language processing to regard each feature of the stop hotspot as a word unit, and reorganize the stop hotspot sequence into a sentence. After that, knowledge transfer is performed by pre-training and fine-tuning to alleviate the problem of low inference accuracy of sparse flow of some data. In the pre-training stage, the present invention proposes an improved BERT model, which uses masked language tasks and two comparative learning tasks to fully capture the stop patterns of truck drivers. In the fine-tuning stage, the present invention adds the ID features of the stop hotspots for similarity calculation to further improve the inference accuracy of the model.
[0062] The present invention provides a method for inferring a stop hotspot sequence based on a BERT language model. Figure 1As shown in the figure, the method is divided into three stages, namely, stop hotspot feature extraction, stop hotspot pre-training representation, and stop hotspot sequence inference. In the stop hotspot feature extraction stage, multiple feature attributes of the stop hotspot are first extracted from a stop hotspot sequence, including the stop hotspot function type, departure time, stop duration, and geographic location; in the stop hotspot pre-training representation stage, the stop hotspot is pre-trained using the improved BERT language model to obtain a high-quality stop hotspot pre-training representation; finally, in the stop hotspot sequence inference stage, the previously pre-trained stop hotspot representation is fine-tuned using the historical data of the current flow direction, and the next stop hotspot that the driver wants to stay is predicted in sequence according to the query request given by the driver until the end is predicted.
[0063] like Figure 1 As shown, the present invention implements a method for inferring a stop hotspot sequence based on a BERT language model, comprising the following steps:
[0064] S1: Stay hotspot attribute extraction stage: Extract multiple characteristic attributes of stay hotspots from the stay hotspot sequence, including stay hotspot type, departure time, stay duration and geographic location.
[0065] In order to fully consider the impact of the stop hotspot function type, time dimension (including departure time, stop duration, etc.) and geospatial dimension (including geospatial location, etc.) on the driver's choice of stop hotspot in transportation tasks, the present invention extracts four attributes, including: 1) the stop hotspot function type (expressed as SH type ): The present invention divides the functional types of the stopover hotspots into five categories (origin, destination, rest area, restaurant and gas station); 2) departure time (expressed as SH departure ): The present invention divides a day [00:00, 24:00) into 24 intervals, namely [00:00, 01:00), [01:00, 02:00), ..., [23:00, 24:00); 3) duration of stay (expressed as SH duration ): The present invention sets three stay time intervals, namely [0 minutes, 40 minutes), [40 minutes, 60 minutes) and [60 minutes, ∞); 4) Geographical space location (expressed as SH space ): In a specific implementation, the map selected is a map of Shandong Province. The present invention divides the map into grids and maps the latitude and longitude coordinates of the hotspots to the grid IDs of the locations of the hotspots. Figure 2As shown, the function type of the last hotspot in the hotspot sequence is destination, the departure time is [18:00, 19:00), the stay duration is [60 minutes, ∞), and the geographic location is grid No. 32 on the map.
[0066] S2: Stay hotspot sequence division phase: The historical stay hotspot sequences are re-divided to align the inputs during training and inference.
[0067] In the embodiment, step S2 specifically includes:
[0068] Since all the stop locations in each transportation task can be known in advance during training, but only the starting point and end point information provided by the driver is available during reasoning, in order to align the inputs during training and reasoning, the present invention re-combines the stop hotspot sequence obtained based on S1, and virtually copies the end point information after the starting point of the stop hotspot sequence used for training, so that the stop hotspot sequence used for training and the sequence input format during reasoning are consistent. At the same time, in order to expand the amount of data for better training and convergence of subsequent models, the present invention performs data enhancement on the combined stop hotspot sequence used for training. Specifically, for a stop hotspot sequence of any length n, the present invention adopts a dynamic interception strategy for data enhancement, that is, each time the first i stop points are extracted (the stop points here include the starting point, the end point and the stop hotspots along the way) as components of the enhanced sequence, where i increases from a minimum value of 3 to a maximum value of n, thereby generating a diverse set of training samples. As Figure 3 As shown, for a stopover hotspot sequence containing three stopover hotspots, the end point is first copied after the start point, and then it is iteratively divided into four new stopover point sequence pairs, where the blue frame in each row represents the stopover hotspot sequence that has been visited, and the purple frame represents the corresponding next stopover hotspot to be visited.
[0069] S3: Stay hotspot sequence embedding stage: Based on the stay hotspot sequence obtained in S2, four learnable encoding layers are used to encode the stay hotspot sequence as the input of the subsequent pre-training model.
[0070] In the embodiment, step S3 specifically includes:
[0071] The present invention constructs each dwell hotspot into a sequence of multiple key-value attribute pairs, where each key or value represents a word element, such as Figure 4As shown, the stop hotspot 0 is represented by 8 words, namely, 'Type', 'Origin', 'Depart', '[11,12)', 'Duration', '[0,40min)', 'Space', and 'grid ID 8'. Its semantics is that the stop hotspot function type of this stop is the starting point, the departure time is between 11 am and 12 pm, the stop duration is between 0 and 40 minutes, and the stop location is located in grid No. 8 of the map. In addition, in addition to the above key-value features, in order to consider the individual differences of drivers, the present invention adds the word 'Driver ID' to distinguish different drivers. Finally, as shown in FIG. Figure 4 As shown, the present invention also adds a 'CLS' word at the beginning of the entire stay hotspot sequence to represent the entire stay hotspot sequence. Then, the present invention uses four learnable encoding layers to encode the input, namely, the stay hotspot position encoding layer E sh , word type encoding layer E tt , word unit position encoding layer E tp and word encoding layer E t . The stop hotspot position encoding layer is used to locate the positions of the word units that make up different stop hotspots, which helps the model learn the sequential dependencies between stop hotspots; the word unit type encoding layer is used to distinguish different types of word units, which helps to learn the influence of different types of features on the next stop hotspot; the word unit position encoding layer is used to locate the positions of different word units, which is similar to the stop hotspot position encoding layer, in order to better capture the sequential dependencies at the word unit level; the word unit encoding layer is used to encode specific word units. Finally, the embedding formula corresponding to the stop hotspot sequence is obtained as follows:
[0072] E emb =E sh +E tt +E tp +E t .
[0073] S4: Pre-training model learning stage: Use the improved BERT language model to pre-train the stay hotspot encoding obtained based on S3 to obtain the stay hotspot pre-training representation.
[0074] In the embodiment, in step S4,
[0075] Considering that the BERT language model has been widely used in the field of natural language processing and has achieved good results, the present invention chooses to use BERT to embed the stay hotspot sequence for further characterization. The calculation formula for BERT to characterize the stay hotspot sequence is as follows:
[0076] [x [CLS] ,x [DRIVER] ,x [emb1] ,...,x [embn] ]=BERT([E[CLS] ,E [DRIVER] ,E [emb1] ,...,E [embn] ]),
[0077] The vector x represents the output of the BERT model. [CLS] ∈R h Used to represent the entire sequence of hotspots, x [DRIVER] represents the representation of the driver word output by the BERT model, x [embn] represents the representation of the nth attribute word output by the BERT model, E [CLS] The embedding vector representing the word ‘CLS’, E [DRIVER] Represents the embedding vector of the driver word, E [embn] represents the embedding vector of the nth attribute word, and h represents the dimension of the embedding.
[0078] At the same time, the present invention adopts the design of masked language tasks in BERT, such as Figure 5 As shown in Figure 1, for a given stop hotspot sequence input, 15% of the tokens are randomly selected for masking, that is, replaced with the '[MASK]' token. Figure 5 In the example, the words 'Origin', '[0,40min)' and 'Space' are masked and replaced with 'Mask'. The loss calculated by the masked language model is as follows:
[0079] p = Softmax(GLEU(W1x i +b1)W2+b2),
[0080]
[0081] Where W1∈R h×h is a learnable parameter, b1∈R h and b2∈R |V| is the bias term, W2∈R h×|V| is the embedding matrix of the dwell hotspot, |V| is the number of dwell hotspots, Softmax() is the softmax activation function, GELU() is the GELU activation function, x i is the representation vector of the i-th word, x i represents the representation vector of each word after BERT, p is the result after softmax function processing, representing the probability of each word being selected, y i is a binary variable. If the word at position i is predicted correctly, then y i =1, otherwise y i =0, p iIt is the predicted probability of the i-th word calculated by the softmax function, which indicates the probability that the model believes that the word at position i belongs to its predicted category.
[0082] In addition, for the contrast learning task between the stop hotspot sequence and the stop hotspot, the purpose of this contrast learning task is to further enhance the sequential dependency between the visited stop hotspot sequence and the next stop hotspot. During training, each sample is recorded as (x seq ,x sh ), where x seq represents the sequence representation of the previously visited hotspots, x sh Represents the next hotspot to be visited. For a training sample in a batch {(x seq1 ,x sh1 ),...,(x seqb ,x shb )},like Figure 5 As shown in the gray background, the present invention combines the representation of the previously visited stop hotspot sequence and the real next stop hotspot representation corresponding to the previously visited stop hotspot sequence representation as a positive sample, and the next stop hotspot representation combination to be visited in other samples as a negative sample. The loss function calculation formula of the comparison learning task between the stop hotspot sequence and the stop hotspot is as follows:
[0083]
[0084]
[0085] Where cos represents the cosine similarity between two vectors, B represents the batch, and τ represents the temperature coefficient.
[0086] The comparative learning task between stay hotspot sequences is to further distinguish stay hotspot sequences with similar stay patterns from stay hotspot sequences with large differences in stay patterns. Specifically, the present invention designs three data enhancement methods: 1) Stay hotspot position offset: randomly select a stay hotspot from the stay hotspot sequence, and randomly select a stay hotspot of the same functional type near its position to replace it; 2) Stay hotspot random masking: randomly select a stay hotspot from the stay hotspot sequence, and use the '[MASK]' word to replace all the words representing the selected stay hotspot; 3) Word random masking: use the '[MASK]' word to randomly mask out the words in the stay hotspot sequence. Figure 5 As shown, during training, given a stay hotspot sequence representation A new representation of the dwell hotspot sequence is obtained by any one or more of the above data enhancement methods. As a positive sample, the stay hotspot sequence representation obtained by enhancing other stay hotspot sequences in the same batch is used as a negative sample. The loss function calculation formula of the comparison learning task between the stay hotspot sequences is as follows:
[0087]
[0088] Where cos represents the cosine similarity between two vectors, B represents the batch, and τ represents the temperature coefficient.
[0089] Finally, the present invention adopts a multi-task learning method to train the model, that is, the loss functions of the masked language model task, the comparison learning task between the dwell hotspot sequence and the dwell hotspot, and the comparison learning task between the dwell hotspot sequences are accumulated as the overall loss function of the pre-trained model. The overall loss function is as follows:
[0090] L all =L mask +αL seq-sh +βL seq-seq ,
[0091] Where α and β are hyperparameters.
[0092] S5: Stay hotspot representation fine-tuning stage: Use the historical data of the current flow direction to fine-tune the stay hotspot pre-trained representation obtained based on S4 to obtain the final stay hotspot sequence prediction model.
[0093] In the embodiment, step S5 specifically includes:
[0094] Taking into account the spatial heterogeneity of the stay patterns of different flow directions, the present invention only uses the data of a single flow direction to fine-tune the pre-trained model obtained based on step S4. In order to enable the model to better capture the higher-level stay patterns between different flow directions, rather than the sequential dependencies between specific stay hotspot IDs, the present invention does not extract the ID features of the stay hotspots in the pre-training stage. However, the fine-tuning stage is only trained on the data set of a single flow direction. At this time, adding the ID features of the stay hotspot can better improve the accuracy of inference; in addition, the contrast learning used in pre-training is performed within a batch, and all negative samples are not taken into account, which inevitably leads to some errors. To this end, in the fine-tuning stage, the present invention first obtains all the stay hotspots of the current fine-tuning flow direction, and inputs them into the trained pre-training model to obtain the representation x of each stay hotspot. sh , and then the embedding layer is used to obtain the representation k of each hotspot ID feature sh , the formula of the fine-tuning model loss function is as follows:
[0095]
[0096] Where M represents the number of hot spots in the current flow, cos represents the cosine similarity between two vectors, B represents the batch, and τ represents the temperature coefficient.
[0097] S6: Stop hotspot sequence prediction stage: Based on the stop hotspot sequence prediction model obtained in S5, the next stop hotspot that the driver wants to stay at is predicted in sequence according to the query request given by the driver until the end point is predicted.
[0098] In the embodiment, step S6 specifically includes:
[0099] In the hotspot sequence prediction stage, the present invention firstly predicts the sequence of all flow direction models MS, the hotspot set SS and the upper limit of the iteration threshold th based on the fine-tuned up , the driver inputs a query request q, where the query request refers to the request issued by the truck driver when performing each transportation task, expressed as q = (Ori, Dest, DestCity, to, td, Dri), where Ori and Dest represent the starting point and the end point respectively, DestCity represents the end point city, to and td represent the departure time and the expected arrival time respectively, and Dri represents the truck driver. Figure 6 As shown, the present invention constructs an original stay hotspot sequence containing only the starting point and the end point according to the Ori and Dest fields in the query request q, and determines the flow direction of the current query according to the DestCity field in the query request q, and finds the fine-tuned model and stay hotspot set corresponding to the flow direction. Then, the original stay hotspot sequence is sent to the fine-tuned model corresponding to the flow direction to obtain the characterization vector of the currently visited stay hotspot sequence, and the similarity between each candidate stay hotspot and the characterization vector of the currently visited stay hotspot sequence is calculated using cosine similarity, and the similarity value is used as the score of the candidate stay hotspot. The stay hotspot with the highest score is inserted into the original stay hotspot sequence to form a new stay hotspot sequence, and then the above steps are performed in sequence until the predicted stay position is the end point or the algorithm reaches the upper limit of iterations.
[0100] In order to verify the effectiveness of the stop hotspot inference framework (hereinafter referred to as SHSI) proposed in this paper, a real data set of a logistics company platform in Shandong was selected. The experimental data is shown in Table 1. The data set includes historical stop hotspot sequence data of bulk freight drivers in four directions, and the seven latest comparison methods in the field of itinerary recommendation are used for experimental verification, including POI-Popularity, POIRank, Markov, DeepTrip, SelftTrip, Query2Trip and BERT-Trip. The evaluation indicators are quantitatively evaluated using F1-score and the longest common subsequence LCSS. The calculation formula of F1-score is as follows:
[0101]
[0102]
[0103]
[0104] TP represents the number of hotspots correctly predicted by the method, FP represents the number of hotspots incorrectly predicted by the method, FN represents the number of hotspots not predicted by the method, Precision represents the consistency between the predicted number of hotspots and the real hotspots, Recall represents how many of the real hotspot data are correctly predicted by the prediction algorithm, and F1-score is the harmonic mean of the above two. However, F1-score does not consider the order relationship between hotspots, so the present invention also uses LCSS evaluation index, that is, experimental evaluation is performed by the longest common subsequence method, for the hotspot sequence shs inferred by the model pred and the real stay hotspot sequence shs real , the calculation formula of LCSS is as follows:
[0105]
[0106] Where l(shs pred ,shs real ) means sh pred and shs real The longest common subsequence of shs, len(shs) represents the number of hotspots contained in the hotspot sequence shs, and the higher the F1-score and LCSS values, the higher the inference accuracy.
[0107] The comparative evaluation results are shown in Table 2, from which it can be seen that SHSI outperforms all benchmark methods in both F1-score and LCSS evaluation indicators, verifying the effectiveness of SHSI. In addition, the effect of the greedy method on some data sets even exceeds the deep method based on traditional machine learning. It is worth noting that not all deep learning-based methods (such as DeepTrip and SelfTrip) are better than traditional machine learning methods (such as POIRank and Markov). The reason may be that the above-mentioned deep learning-based models do not fully capture the stay patterns of truck drivers. BERT-Trip achieved better results. Except for the SHSI proposed in the present invention, BERT-Trip is more accurate than all other benchmark methods. The reason may be that the masked language task of BERT is used, which enables the model to encode the context bidirectionally, thereby better capturing the driver's stay pattern.
[0108] Table 1 Dataset description
[0109]
[0110] Table 2 Comparison of the effects of different methods
[0111]
[0112] In summary, the present invention comprehensively adopts the BERT language model and contrastive learning and other technologies to form a freight stop hotspot sequence inference method based on the BERT language model. Through trajectory data mining, it can accurately infer the stop hotspot sequence where the truck driver wants to stop, thereby providing effective assistance for route planning and transportation time prediction.
[0113] References
[0114] [1]CHEN D,ONG CS,XIE L.Learning points and routes to recommend trajectories[C] / / Proceedings of the 25th ACM international on conference oninformation and knowledge management.2016:2227–2232.
[0115] [2]HE J,QI J,RAMAMOHANARAO KA joint context-aware embedding for triprecommendations[C] / / 2019IEEE 35th International Conference on DataEngineering(ICDE).:IEEE,2019:292–303.
[0116] [3]GAO Q,TRAJCEVSKI G,ZHOU F,et al.Deeptrip: Adversarially understanding human mobility for trip recommendation[C] / / Proceedings of the27th ACM SIGSPATIAL international conference on advances in geographicinformation systems.2019:444–447.
[0117] [4]HUANG J,LIU Y,CHENY,et al.Dynamic recommendation ofpoi sequenceresponding to historical trajectory[J].ISPRS International Journal ofGeo-Information,2019,8(10):433.
[0118] [5]GAO Q,WANG W,ZHANG K,et al.Self-supervised representation learning for trip recommendation[J].Knowledge-Based Systems,2022,247:108791.
[0119] [6]KUO AT,CHEN H,KU W S.Bert-trip:Effective and scalable triprepresentation using attentive contrast learning[C] / / 2023IEEE 39thInternational Conference on Data Engineering(ICDE).:IEEE Computer Society, 2023:612–623.
[0120] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.
Claims
1. A method for inferring freight stop hotspot sequences based on the BERT language model, characterized in that: The following steps are involved: S1: Stay hotspot attribute extraction stage: extract the characteristic attributes of stay hotspots from the stay hotspot sequence; S2: Stay hotspot sequence division stage: re-dividing the historical stay hotspot sequence to align the inputs during training and inference; S3: Stay hotspot sequence embedding stage: Based on the stay hotspot sequence obtained in S2, four learnable encoding layers are used to encode the stay hotspot sequence as the input of the subsequent pre-training model; S4: Pre-training model learning stage: The improved BERT language model after introducing two contrastive learning proxy tasks is used to pre-train the stay hotspot encoding obtained based on S3 to obtain the stay hotspot pre-training representation; S5: fine-tuning of the representation of the stay hotspot: the historical data of the current flow direction is used to fine-tune the pre-trained representation of the stay hotspot obtained based on S4 to obtain the final stay hotspot sequence prediction model; S6: Stay hotspot sequence prediction stage: Based on the final stay hotspot sequence prediction model obtained in S5, the next stay hotspot is predicted in sequence according to the given query request until the end is predicted and the process is terminated.
2. The method for inferring freight stop hotspot sequences according to claim 1, characterized in that: In step S1, the characteristic attributes of the hotspot include the hotspot function type, departure time, stay duration and geographic location; wherein, The stopover hotspot function types include starting point, end point, rest area, restaurant and gas station; The departure time period refers to dividing a day [00:00, 24:00) into 24 intervals, including [00:00, 01:00), [01:00, 02:00), ..., [23:00, 24:00); The stay duration refers to the three set stay duration intervals, which are [0 minutes, 40 minutes), [40 minutes, 60 minutes) and [60 minutes, ∞); The geographic space location refers to dividing a map into grids, and mapping the latitude and longitude coordinates of the stay hotspot to the grid ID of the location of the stay hotspot.
3. The method for inferring freight stop hotspot sequences according to claim 1, characterized in that: In step S2, The stay hotspot sequences obtained in step S1 are combined, and the end point information is virtually copied after the starting point of the stay hotspot sequence for training, so that the stay hotspot sequence for training and the sequence input format during inference are consistent; and a dynamic interception strategy is used to perform data enhancement on the stay hotspot sequence for training. In a stay hotspot sequence of length n, the first i stay points are extracted each time as components of the enhanced sequence, where i increases from a minimum value of 3 to a maximum value of n, to generate a training sample set.
4. The method for inferring a freight stop hotspot sequence according to claim 1, characterized in that: In step S3, Construct each stop hotspot into a sequence of one or more key-value attribute pairs, where each key or value represents a word; add a 'Driver ID' word to distinguish different drivers and a 'CLS' word to represent the entire stop hotspot sequence; and / or, Encode layer E with the dwell hotspot locations sh , word type encoding layer E tt , word unit position encoding layer E tp and word encoding layer E t The entire stop hotspot sequence is encoded; wherein the stop hotspot position encoding layer is used to locate the positions of the word units that constitute different stop hotspots, so that the model learns the sequential dependency between stop hotspots; the word unit type encoding layer is used to distinguish different types of word units, so that the model learns the influence of different types of features on the next stop hotspot; the word unit position encoding layer is used to locate the positions of different word units, so that the model learns the sequential dependency at the word unit level; the word unit encoding layer is used to encode specific word units; finally, the embedding formula corresponding to the stop hotspot sequence is obtained as follows: AND emb =And sh +E tt +E tp +E t 。 5. The method for inferring freight stop hotspot sequences according to claim 1, characterized in that: In the step S4, BERT is used to further characterize the dwell hotspot sequence embedding. The specific calculation formula of the BERT language model is as follows: [x [CLS] ,x [DRIVER] ,x [emb1] ,...,x [embo] ]=BERT([E [CLS] ,AND [DRIVER] ,AND [emb1] ,...,AND [embn] ]), Among them, vector x represents the output of the BERT model, x [CLS] ∈R h represents the characterization of the entire sequence of hotspots, x [DRIVER] represents the representation of the driver word output by the BERT model, x [embn] represents the representation of the nth attribute word output by the BERT model, E [CLS] The embedding vector representing the word ‘CLS’, E [DRIVER] Represents the embedding vector of the driver word, E [embn] represents the embedding vector of the nth attribute word, h represents the dimension of the embedding; and / or, Through the masked language task in BERT, for a given stop hotspot sequence input, 15% of the tokens are randomly selected for masking, that is, replaced with '[MASK]' tokens; the loss calculated by the masked language model is expressed as follows: p=Softmax(GLEU(W1x i +b1)W2+b2), Where W1∈R h×h is a learnable parameter, b1∈R h and b2∈R |V| is the bias term, W2∈R h×|V| is the embedding matrix of the dwell hotspot, |V| is the number of dwell hotspots, Softmax() is the softmax activation function, GELU() is the GELU activation function, x i is the representation vector of the i-th word, x i represents the representation vector of each word after BERT, p is the result after softmax function processing, representing the probability of each word being selected, y i is a binary variable. If the word at position i is predicted correctly, then y i =1, otherwise y i =0, p i It is the predicted probability of the i-th word calculated by the softmax function, which indicates the probability that the model believes that the word at position i belongs to its predicted category.
6. The method for inferring freight stop hotspot sequences according to claim 5, characterized in that: In the step S4, For the learning task of contrasting the sequence of stay hot spots and the stay hot spots, during training, each sample is recorded as (x seq ,x sh ), where x seq represents the sequence representation of the previously visited hotspots, x sh Represents the next hotspot representation to be visited; for a training sample in a batch {(x seq1 ,x sh1 ),...,(x seqb ,x shb )}, the previously visited stop hotspot sequence representation and the real next stop hotspot representation corresponding to the previously visited stop hotspot sequence representation are combined as positive samples, and the next stop hotspot representation combination to be visited in other samples is combined as negative samples; the loss function calculation formula of the stop hotspot sequence and stop hotspot comparison learning task is as follows: For the comparative learning task between stay hotspot sequences, three data enhancement methods are used: stay hotspot position offset, stay hotspot random mask, and word unit random mask. During training, given a stay hotspot sequence representation A new sequence representation of the dwell hotspots is obtained by any one or more of the data enhancement methods. As positive samples, the representation of the stay hotspot sequence obtained by enhancing other stay hotspot sequences in the same batch is used as a negative sample. The loss function calculation formula for the comparative learning task between stay hotspot sequences is as follows: Where cos represents the cosine similarity between two vectors, B represents the batch, and τ represents the temperature coefficient; The loss functions of the masked language model task, the contrastive learning task between the dwelling hotspot sequence and the dwelling hotspot, and the contrastive learning task between the dwelling hotspot sequences are accumulated as the overall loss function of the pre-training model. The overall loss function is as follows: L all =L mask +αL seq-sh +βL seq-seq , Where α and β are hyperparameters.
7. The method for inferring freight stop hotspot sequences according to claim 1, characterized in that: In step S5, Using the data of a single flow direction, the ID feature of the stay hotspot is added to fine-tune the pre-trained model obtained based on step S4: first, all the stay hotspots of the current fine-tuned flow direction are obtained, and all the stay hotspots of the fine-tuned flow direction are input into the trained pre-trained model to obtain the representation x of each stay hotspot sh , and then the embedding layer is used to obtain the representation k of each hotspot ID feature sh , the formula of the fine-tuning model loss function is as follows: Where M represents the number of hot spots in the current flow, cos represents the cosine similarity between two vectors, B represents the batch, and τ represents the temperature coefficient.
8. The method for inferring freight stop hotspot sequences according to claim 1, characterized in that: In step S6, In the hotspot sequence prediction stage, firstly, based on the fine-tuned set of all flow models MS, the hotspot set SS and the upper limit of the iteration threshold th up , input the query request q, construct the original hotspot sequence containing only the starting point and the end point, and determine the flow direction of the current query according to the end city field in the query request q, and find the fine-tuned model and hotspot set corresponding to the flow direction; send the original hotspot sequence to the fine-tuned model of the corresponding flow direction, use cosine similarity to calculate the score of each candidate hotspot, insert the hotspot with the highest score into the original hotspot sequence to form a new hotspot sequence, and then perform the above steps in sequence until the predicted stop position is the end point or the algorithm reaches the iteration limit.
Citation Information
Patent Citations
Transportation terminal point calibration method and system for bulk logistics
CN115423841A
Transportation arrival time length prediction method for bulk logistics
CN117726058A
