A sub-trajectory clustering method and device based on reinforcement learning

Abstract

One or more embodiments of the present specification provide a sub-trajectory clustering method and apparatus based on reinforcement learning. The method includes: obtaining a trajectory set to be divided, wherein the trajectory set includes at least one trajectory, and the at least one trajectory includes k trajectories used as initial clustering centers; performing reinforcement learning on the at least one trajectory, traversing each trajectory point included in each trajectory to determine a division action corresponding to each trajectory point, and using the division action corresponding to each trajectory point to divide the trajectory where each trajectory point is located to obtain at least one sub-trajectory; clustering each of the divided sub-trajectories to obtain k sub-trajectory sets, so that the values ​​of the clustering evaluation indicators of the k sub-trajectory sets obtained by clustering meet the expected values; and obtaining the k sub-trajectory sets obtained based on reinforcement learning as the sub-trajectory clustering results for the trajectory set.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of computer application technology, and more particularly, to a sub-trajectory clustering method and apparatus based on reinforcement learning. Background Art

[0002] With the rapid development of communication networks, satellite positioning, and mobile internet technologies, as well as the widespread use of mobile communication terminals, the spatiotemporal data generated by various mobile objects, such as pedestrians, vehicles, and aircraft, has exploded. Mining and analyzing the trajectory data of mobile objects has become a significant research issue. Trajectory clustering is a hot topic in this area.

[0003] In practical applications, the entire trajectory can be divided into one or more sub-trajectories according to manually formulated trajectory division rules, and then similar sub-trajectories can be clustered into the same cluster based on the similarity between the divided sub-trajectories.

[0004] It can be seen from this that, on the one hand, for different application scenarios, corresponding trajectory division rules need to be manually formulated, resulting in poor versatility of trajectory division rules and high technical requirements for users who formulate trajectory division rules; on the other hand, the two processes of trajectory division and sub-trajectory clustering are implemented independently. The quality of trajectory division results may affect the quality of subsequent sub-trajectory clustering results. Summary of the Invention

[0005] The present application provides a sub-trajectory clustering method based on reinforcement learning, the method comprising:

[0006] Obtaining a trajectory set to be divided; wherein the trajectory set includes at least one trajectory; the at least one trajectory includes k trajectories used as initial cluster centers, where k is a positive integer; and the trajectory is a trajectory point sequence including at least one trajectory point;

[0007] performing reinforcement learning on the at least one trajectory, traversing each trajectory point included in each trajectory to determine a partitioning action corresponding to each trajectory point, partitioning the trajectory containing each trajectory point using the partitioning action corresponding to each trajectory point to obtain at least one sub-trajectory, and clustering each of the sub-trajectories obtained by the partitioning to obtain k sub-trajectory sets, such that a clustering evaluation index of each of the k sub-trajectory sets obtained by clustering meets an expected value;

[0008] K sub-trajectory sets obtained based on reinforcement learning are obtained as sub-trajectory clustering results for the trajectory set.

[0009] Optionally, the decision-making action of the reinforcement learning includes: a division action of determining whether to divide the trajectory where the traversed trajectory point is located; wherein the division action includes a first division action of dividing the trajectory where the trajectory point is located, and a second division action of not dividing the trajectory where the trajectory point is located;

[0010] The reward corresponding to the reinforcement learning decision action includes: the value of a clustering evaluation index for the k sub-trajectories obtained by clustering the sub-trajectories obtained by dividing the trajectory containing the traversed trajectory point using the partitioning action corresponding to the traversed trajectory point; wherein the clustering evaluation index includes the average distance between each sub-trajectory and the cluster center of the sub-trajectory set to which it belongs.

[0011] Optionally, the values ​​of the clustering evaluation index of the k sub-trajectory sets meet expected values, including: the average value of the distances from each sub-trajectory to the cluster center of the sub-trajectory set to which it belongs is the smallest;

[0012] The clustering evaluation index is characterized by the following formula:

[0013]

[0014]

[0015] Among them, OD represents the clustering evaluation index of the k sub-trajectory set; m represents the total number of sub-trajectories included in the k sub-trajectory set; m i Represents the i-th sub-trajectory set C i The number of sub-trajectories included in OD i Represents the i-th sub-trajectory set C i Clustering evaluation index; x represents the set of sub-trajectories belonging to the i-th sub-trajectory set C i The sub-trajectory included in c i Represents the i-th sub-trajectory set C i The cluster center of dist(x,c i ) represents the sub-trajectory x to the sub-trajectory set C to which it belongs i The cluster center c i distance.

[0016] Optionally, the cumulative reward of the reinforcement learning includes: the sum of the instantaneous rewards corresponding to each traversed trajectory point;

[0017] The instant reward corresponding to the traversed trajectory point includes: the difference between the first value and the second value of the clustering evaluation index corresponding to the traversed trajectory point; wherein, the first value includes the value of the clustering evaluation index before the division action corresponding to the traversed trajectory point is adopted to divide the trajectory where the trajectory point is located; the second value includes the value of the clustering evaluation index after the division action corresponding to the traversed trajectory point is adopted to divide the trajectory where the trajectory point is located, and clustering the sub-trajectories obtained by the division.

[0018] Optionally, the environmental state related to the decision-making action of the reinforcement learning includes: a third value and a fourth value of the clustering evaluation index corresponding to the traversed trajectory point; wherein the third value includes the value of the clustering evaluation index after the traversed trajectory point is divided by the first division action, and each sub-trajectory obtained by the division is clustered; the fourth value includes the value of the clustering evaluation index after the traversed trajectory point is not divided by the second division action, and each sub-trajectory obtained by the division is clustered.

[0019] Optionally, the environmental state related to the decision-making action of the reinforcement learning further includes: one or more of: a baseline value of the cluster evaluation index, a first ratio corresponding to the traversed trajectory point, and a second ratio corresponding to the traversed trajectory point;

[0020] Wherein, the benchmark value includes the preset value of the cluster evaluation index;

[0021] The first ratio includes a ratio of a first trajectory length corresponding to the traversed trajectory point to a length of a trajectory in which the traversed trajectory point is located; the first trajectory length includes the number of trajectory points between the traversed trajectory point and a previous trajectory point in the trajectory in which the traversed trajectory point is located, the corresponding division action of which is the first division action; the length of the trajectory in which the traversed trajectory point is located includes the number of trajectory points in the trajectory in which the traversed trajectory point is located;

[0022] The second ratio includes the ratio of the second trajectory length corresponding to the traversed trajectory point to the length of the trajectory where the traversed trajectory point is located; the second trajectory length includes the number of trajectory points between the traversed trajectory point and the last trajectory point included in the trajectory where the traversed trajectory point is located.

[0023] Optionally, the performing reinforcement learning on the at least one trajectory, traversing each trajectory point included in each trajectory to determine a division action corresponding to each trajectory point, includes:

[0024] Traversing each trajectory point included in each trajectory;

[0025] The environmental state corresponding to the traversed trajectory point is used as input data and input into a trained reinforcement learning-based decision model to obtain a first decision value corresponding to the first division action and a second decision value corresponding to the second division action output by the decision model; wherein the first decision value is used to represent the maximum value of the cumulative reward corresponding to the first division action under the environmental state corresponding to the traversed trajectory point; and the second decision value is used to represent the maximum value of the cumulative reward corresponding to the second division action under the environmental state corresponding to the traversed trajectory point;

[0026] If the first decision value is greater than the second decision value, determining the division action corresponding to the traversed trajectory point as the first division action;

[0027] If the first decision value is not greater than the second decision value, the division action corresponding to the traversed trajectory point is determined to be the second division action.

[0028] Optionally, the decision model includes: an MDP model based on a neural network.

[0029] Optionally, obtaining k sub-trajectory sets obtained based on reinforcement learning as sub-trajectory clustering results for the trajectory set includes:

[0030] Based on the k sub-trajectory sets obtained by clustering, update the cluster center of each sub-trajectory set;

[0031] If the distance between the updated cluster center and the pre-updated cluster center of each sub-trajectory set does not exceed a preset threshold, the iterative process of the reinforcement learning is stopped, and the k sub-trajectory sets obtained by clustering are obtained as the sub-trajectory clustering result for the trajectory set.

[0032] Optionally, clustering the sub-trajectories obtained by the division to obtain k sub-trajectory sets includes:

[0033] Calculate the distance between each sub-trajectory obtained by division and the k cluster centers;

[0034] Each sub-trajectory is divided into the sub-trajectory set corresponding to the cluster center closest to it.

[0035] The present application also provides a sub-trajectory clustering system based on reinforcement learning, which is used to implement the above method.

[0036] The present application also provides an electronic device, comprising a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus;

[0037] The memory stores machine-readable instructions, and the processor executes the above method by calling the machine-readable instructions.

[0038] The present application also provides a machine-readable storage medium, which stores machine-readable instructions. When the machine-readable instructions are called and executed by a processor, the above method is implemented.

[0039] Through the above embodiments, on the one hand, by treating the trajectory segmentation process as a sequential decision-making process, reinforcement learning can be performed on at least one trajectory included in the trajectory set (i.e., a sequence of trajectory points), traversing each trajectory point included in each trajectory to determine the segmentation action corresponding to each trajectory point. This can adapt to the trajectory segmentation requirements of the sub-trajectory clustering problem in different application scenarios based on the intrinsic characteristics of the different trajectories to be segmented, avoiding the manual formulation of different trajectory segmentation rules, and can be completely data-driven, making it more universal. On the other hand, solving the sub-trajectory clustering problem based on reinforcement learning can use the sub-trajectory clustering results to guide the trajectory segmentation process, thereby learning the trajectory segmentation decision that can produce the optimal sub-trajectory clustering results, and ultimately obtaining a more optimal sub-trajectory clustering result. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 is a schematic diagram of a reinforcement learning algorithm shown in an exemplary embodiment;

[0042] Figure 2 is a flowchart of a sub-trajectory clustering method based on reinforcement learning, shown in an exemplary embodiment;

[0043] Figure 3 is a schematic diagram of trajectory division shown in an exemplary embodiment;

[0044] Figure 4 is a flowchart of another sub-trajectory clustering method based on reinforcement learning, shown in an exemplary embodiment;

[0045] Figure 5 is a schematic diagram of distance calculation shown in an exemplary embodiment;

[0046] Figure 6 is a schematic structural diagram of an electronic device in which a sub-trajectory clustering device based on reinforcement learning is located, showing an exemplary embodiment;

[0047] Figure 7 It is a block diagram of a sub-trajectory clustering device based on reinforcement learning, showing an exemplary embodiment. DETAILED DESCRIPTION

[0048] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative work should fall within the scope of protection of this specification.

[0049] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0050] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of this specification, the following briefly describes the related technologies of sub-trajectory clustering involved in the embodiments of this specification.

[0051] With the rapid development of communication networks, satellite positioning, and mobile internet technologies, as well as the widespread use of mobile communication terminals, the spatiotemporal data generated by various mobile objects, such as pedestrians, vehicles, and aircraft, has exploded. Mining and analyzing the trajectory data of mobile objects has become a significant research issue. Trajectory clustering is a hot topic in this area.

[0052] Trajectory clustering involves segmenting a set of trajectories into clusters based on a specific similarity criterion (e.g., distance). This ensures that trajectories in the same cluster are as similar as possible, while trajectories in different clusters are as similar as possible. Trajectory clustering can identify similar trajectories, playing an important role in location-based services, motion behavior research, mobile object tracking, hotspot prediction, and abnormal activity detection.

[0053] In practical applications, since trajectory clustering is performed based on the entire trajectory as the basic unit, local features of complex trajectories may be ignored, making it difficult to discover common sub-trajectory patterns. Therefore, the sub-trajectory clustering problem can be studied. That is, the entire trajectory can be divided into one or more sub-trajectories according to manually formulated division rules. Then, similar sub-trajectories can be clustered into the same cluster based on the similarity between the divided sub-trajectories.

[0054] It can be seen that in the embodiments shown above, on the one hand, it is necessary to manually formulate corresponding trajectory division rules for different application scenarios, resulting in poor versatility of the trajectory division rules and high technical requirements for users who formulate the trajectory division rules; on the other hand, the two processes of trajectory division and sub-trajectory clustering are implemented independently. The quality of the trajectory division result may affect the quality of the sub-trajectory clustering result obtained by subsequent clustering based on the trajectory division result.

[0055] For example, for multiple trajectories that include a portion that "first moves at a uniform speed and then decelerates," if the trajectories are divided according to changes in speed, the uniform motion portion and the deceleration portion of the trajectories will be divided into different sub-trajectories. Furthermore, when clustering the resulting sub-trajectories, at least two clusters will be obtained. That is, the sub-trajectories corresponding to the uniform motion portion of the trajectories and the sub-trajectories corresponding to the deceleration portion of the trajectories will be divided into different clusters. In fact, the "first moves at a uniform speed and then decelerates" portion of the trajectories can be directly divided into a sub-trajectory. Then, when clustering the resulting sub-trajectories, the sub-trajectories corresponding to the "first moves at a uniform speed and then decelerates" portion of the trajectories can be grouped into the same cluster. This allows us to discover that the common local pattern of the multiple trajectories is "first moves at a uniform speed and then decelerates." Based on this, in the field of traffic prediction, based on the above sub-trajectory clustering results, it can be analyzed that traffic congestion may have occurred on the road section where the multiple trajectories are located.

[0056] In view of this, this specification aims to propose a sub-trajectory clustering method based on reinforcement learning (RL), which can use the sub-trajectory clustering results to guide the trajectory division process, thereby obtaining a trajectory division scheme that can make the sub-trajectory clustering results better.

[0057] Reinforcement learning is a field in machine learning, which can also be called reinforcement learning, evaluation learning, or enhanced learning. Figure 1 , Figure 1This is a schematic diagram of a reinforcement learning algorithm, illustrating an exemplary embodiment. Reinforcement learning can simulate the biological evolution process in nature. After an agent takes a specific action, it receives a reward from the environment and observes a new state. Furthermore, the agent can take the next action based on the reward and the observed state, thus forming a closed loop. Ultimately, the agent learns an optimal strategy, guiding the agent's decision-making actions in a specific environment to maximize the cumulative reward.

[0058] During implementation, a set of trajectories to be divided can be obtained; wherein, the trajectory set includes at least one trajectory; the at least one trajectory includes k trajectories used as initial clustering centers, where k is a positive integer; the trajectory is a trajectory point sequence including at least one trajectory point; further, reinforcement learning can be performed on the at least one trajectory, and each trajectory point included in each trajectory is traversed to determine a division action corresponding to each trajectory point, and the trajectory where each trajectory point is located is divided using the division action corresponding to each trajectory point to obtain at least one sub-trajectory, and each sub-trajectory obtained by the division is clustered to obtain k sub-trajectory sets, so that the value of the clustering evaluation index of the k sub-trajectory sets obtained by clustering meets the expected value; further, k sub-trajectory sets obtained based on reinforcement learning can be obtained as the sub-trajectory clustering result for the trajectory set.

[0059] Thus, it can be seen that in the technical solution of this specification, on the one hand, by treating the trajectory division process as a sequential decision process, reinforcement learning can be performed on at least one trajectory included in the trajectory set (that is, a sequence of trajectory points), traversing each trajectory point included in each trajectory to decide the division action corresponding to each trajectory point, thereby being able to adapt to the trajectory division requirements of the sub-trajectory clustering problem in different application scenarios according to the intrinsic characteristics of the different trajectories to be divided, avoiding the manual formulation of different trajectory division rules, and can be completely data-driven, making it more universal. On the other hand, based on reinforcement learning to solve the sub-trajectory clustering problem, the sub-trajectory clustering results can be used to guide the trajectory division process, thereby learning the trajectory division decision that can produce the optimal sub-trajectory clustering result, and ultimately obtaining a better sub-trajectory clustering result.

[0060] The present application is described below through specific embodiments and in combination with specific application scenarios.

[0061] See Figure 2 , Figure 2This is a flowchart of a sub-trajectory clustering method based on reinforcement learning, which is shown in an exemplary embodiment. The method can perform the following steps:

[0062] Step 202: Obtain a trajectory set to be divided; wherein the trajectory set includes at least one trajectory; the at least one trajectory includes k trajectories used as initial cluster centers, where k is a positive integer; and the trajectory is a trajectory point sequence including at least one trajectory point.

[0063] For example, the set of trajectories to be divided {T1, T2, ..., T M}, where T i The i-th trajectory in the trajectory set may be represented by M, where M is a positive integer and the trajectory set includes M trajectories. The M trajectories may include k trajectories used as initial cluster centers, where k is a positive integer.

[0064] As an exemplary description, the trajectory T1 can be represented as a trajectory point sequence, T1= <p1,p2,...,p n >, where p i It can represent the i-th track point in the track T1, n is a positive integer, and the track point sequence includes n track points. Specifically, the track point p i It can be represented as (lat i ,lon i , t i ), where lat i ,lon i The trajectory point p can be represented by i The spatial position, that is, the longitude and latitude coordinates, t i The trajectory point p can be represented by i The sampling time.

[0065] It should be noted that this specification does not limit the specific method of determining the initial cluster centers. For example, k can be the number of pre-configured cluster centers, and k trajectories can be randomly selected from the acquired set of trajectories to be divided as the initial cluster centers; wherein the cluster centers can be represented by c1, c2, ..., c k .

[0066] Step 204: Reinforcement learning is performed on the at least one trajectory, and each trajectory point included in each trajectory is traversed to determine a partitioning action corresponding to each trajectory point. The trajectory where each trajectory point is located is partitioned using the partitioning action corresponding to each trajectory point to obtain at least one sub-trajectory. Each sub-trajectory obtained by the partitioning is clustered to obtain k sub-trajectory sets, so that the values ​​of the clustering evaluation indicators of the k sub-trajectory sets obtained by clustering meet the expected values.

[0067] For example, after obtaining the trajectory set to be divided {T1, T2, ..., T M After that, reinforcement learning can be performed on the M trajectories included in the trajectory set, and each trajectory point included in each trajectory can be traversed to determine the division action corresponding to each trajectory point. The trajectory where each trajectory point is located can be divided using the division action corresponding to each trajectory point to obtain at least one sub-trajectory, and each sub-trajectory obtained by division can be clustered to obtain k sub-trajectory sets C1, C2, ..., C k , so that the values ​​of the clustering evaluation index of the k sub-trajectory sets obtained by clustering meet the expected values.

[0068] Wherein, a sub-trajectory can be a subset of continuous trajectory points in a trajectory. As an exemplary description, a sub-trajectory of the trajectory T1 can be represented as sub_T1(i, j)= <p i , p t+1 ,...,p j >, 1≤i≤j≤n, the sub-trajectory sub_T1(i, j) can represent the sub-trajectory obtained by dividing the trajectory T1 at the positions of the i-th trajectory point and the j-th trajectory point in the trajectory T1.

[0069] In step 204, the values ​​of the clustering evaluation index for the k sub-trajectory sets meet the expected values, and those skilled in the art can flexibly set them as needed. For example, the clustering evaluation index can include the average distance between each sub-trajectory and the cluster center of the sub-trajectory set to which it belongs. Then, the values ​​of the clustering evaluation index for the k sub-trajectory sets meet the expected values, which can mean that the value of the clustering evaluation index for the k sub-trajectory sets is the smallest, that is, the average value of the distances between each sub-trajectory obtained by division and the cluster center of the sub-trajectory set to which it belongs is the smallest.

[0070] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of this specification, the decision-making actions, rewards, and environmental states that may be involved in the reinforcement learning are first described below.

[0071] In one embodiment shown, in step 204, the decision-making action of the reinforcement learning may specifically include: a division action of whether to divide the trajectory where the traversed trajectory point is located; wherein the division action may include a first division action of dividing the trajectory where the trajectory point is located, and a second division action of not dividing the trajectory where the trajectory point is located.

[0072] For example, the first division action and the second division action can be represented as 1 and 0 respectively; if the decision is made that the trajectory point p i The corresponding division action is 1, then at the trajectory point p i At the location of the trajectory point p i The trajectory T1 is divided, and the new sub-trajectories obtained by the division are clustered to cluster the new sub-trajectories into a certain sub-trajectory set; if the decision is made that the trajectory point p i The corresponding division action is 0, then the trajectory point p i At the location where the trajectory point p is located, i The trajectory T1 is divided.

[0073] In one embodiment shown, in step 204, the reward corresponding to the reinforcement learning decision action may specifically include: the value of a clustering evaluation index for a set of k sub-trajectories obtained by clustering the sub-trajectories obtained by dividing the trajectory containing the traversed trajectory point using a division action corresponding to the traversed trajectory point; wherein the clustering evaluation index may include the average distance between each sub-trajectory and the cluster center of the sub-trajectory set to which it belongs.

[0074] In this case, in step 204 , the values ​​of the clustering evaluation indicators of the k sub-trajectory sets meet expected values, which may include: the average value of the distances from each sub-trajectory to the cluster center of the sub-trajectory set to which it belongs is the smallest.

[0075] For example, the cluster evaluation index OD can be represented by the following formula:

[0076]

[0077]

[0078] Among them, OD represents the clustering evaluation index of the k sub-trajectory set; m represents the total number of sub-trajectories included in the k sub-trajectory set; m i Represents the i-th sub-trajectory set C i The number of sub-trajectories included in OD i Represents the i-th sub-trajectory set C i Clustering evaluation index; x represents the set of sub-trajectories belonging to the i-th sub-trajectory set C i The sub-trajectory included in c i Represents the i-th sub-trajectory set C i The cluster center of dist(x,c i ) represents the sub-trajectory x to the sub-trajectory set C to which it belongs i The cluster center c i distance.

[0079] It should be noted that in the above embodiment, the smaller the value of OD is, the smaller the average distance between each sub-trajectory included in the k sub-trajectory sets obtained by clustering and the cluster center of the sub-trajectory set to which it belongs is, that is, the better the sub-trajectory clustering result for the trajectory set is.

[0080] In one embodiment shown, in step 204, the cumulative reward of the reinforcement learning may specifically include: the sum of the immediate rewards corresponding to each traversed trajectory point. The immediate reward corresponding to the traversed trajectory point may specifically include: the difference between a first value and a second value of the clustering evaluation index corresponding to the traversed trajectory point; wherein the first value includes the value of the clustering evaluation index before the trajectory containing the traversed trajectory point is divided by the division action corresponding to the traversed trajectory point; and the second value includes the value of the clustering evaluation index after the trajectory containing the traversed trajectory point is divided by the division action corresponding to the traversed trajectory point, and the sub-trajectories obtained by the division are clustered.

[0081] For example, with the traversed trajectory point p i Corresponding immediate reward i It can be characterized as:

[0082] reward i =s i-1 ·OD-s i ·OD

[0083] Among them, s i-1 OD can represent the first value of the cluster evaluation index, that is, when using the trajectory point p i The corresponding division action is for the trajectory point p i The value of the clustering evaluation index OD before the trajectory is divided; s i OD can represent the second value of the cluster evaluation index, that is, using the value of the trajectory point p i The corresponding division action is for the trajectory point p i The trajectory is divided, and the value of the clustering evaluation index OD is obtained after clustering each sub-trajectory obtained by the division.

[0084] Accordingly, the cumulative reward REWARD of the reinforcement learning can be characterized as:

[0085] REWARD=reward1+reward2+…+reward n

[0086] =s0·OD-s1·OD+s1·OD-s2·OD+…+s n-1 ·OD-sn ·OD

[0087] =s0·OD-s n ·OD

[0088] Among them, s0·OD can represent the initial value of the clustering evaluation index OD before dividing each trajectory, which is a fixed value. It can be seen that the learning goal of reinforcement learning can be to maximize the value of the cumulative reward, that is, to make s n The OD value is the smallest.

[0089] In one embodiment shown, in step 204, the environmental state related to the decision-making action of the reinforcement learning may specifically include: a third value and a fourth value of the clustering evaluation index corresponding to the traversed trajectory point; wherein the third value includes the value of the clustering evaluation index after the trajectory of the traversed trajectory point is divided by the first division action, and each sub-trajectory obtained by the division is clustered; the fourth value includes the value of the clustering evaluation index after the trajectory of the traversed trajectory point is not divided by the second division action, and each sub-trajectory obtained by the division is clustered.

[0090] For example, the environment state state of the reinforcement learning can be represented as: state = (OD s , OD n ), among which, OD s It can represent the third value of the clustering evaluation index, that is, the value of the clustering evaluation index OD after the first division action is used to divide the trajectory where the traversed trajectory point is located and clustering the sub-trajectories obtained by the division; OD n It can represent the fourth value of the clustering evaluation index, that is, the value of the clustering evaluation index OD after the second division action is used to not divide the trajectory where the traversed trajectory point is located and clustering the sub-trajectories obtained by the division.

[0091] In order to make those skilled in the art better understand the technical solutions in the embodiments of this specification, Figure 3 , an example is given to illustrate the specific implementation method of determining the first value, second value, third value and fourth value of the cluster evaluation index.

[0092] See Figure 3 , Figure 3 FIG. 1 is a schematic diagram showing a trajectory division according to an exemplary embodiment. Figure 3What is shown is a part of the trajectory T1. The division action corresponding to the first trajectory point p1 in the trajectory T1 can be defaulted to the first division action. Assuming that the division action corresponding to the trajectory point p2 in the trajectory T1 is the second division action, and the division action corresponding to the trajectory point p3 in the trajectory T1 is the first division action, when traversing to the trajectory point p3, the trajectory T1 can be divided at the position of the trajectory point p3 to obtain a new sub-trajectory sub_T1(1,3), and clustering is performed on the sub-trajectory sub_T1(1,3). Mindist1 can be used to represent the distance from the sub-trajectory sub_T1(1,3) to the cluster center of the sub-trajectory set described therein, and num can be used to represent the current number of sub-trajectories. Then Furthermore, we can traverse to the trajectory point p4, then the first value of the clustering index is s3·OD; assuming that it is decided to divide the trajectory T1 at the trajectory point p4, a new sub-trajectory sub_T1(3, 4) can be obtained. Mindist2 can be used to represent the distance from the sub-trajectory sub_T1(1, 3) to the cluster center of the sub-trajectory set described above, and num+1 can be used to represent the current number of sub-trajectories. The third value of the clustering index can be expressed as The second value of the clustering index can be expressed as s4·OD=OD s Assuming that it is decided not to divide the trajectory T1 at the trajectory point p4, the fourth value of the clustering index can be expressed as The second value of the clustering index can be expressed as s4·OD=OD n .

[0093] In another embodiment shown, in step 204, the environmental state related to the decision-making action of the reinforcement learning may further include: one or more of: a baseline value of the cluster evaluation index, a first ratio corresponding to the traversed trajectory point, and a second ratio corresponding to the traversed trajectory point; wherein the baseline value may include a preset value of the cluster evaluation index; the first ratio may include a ratio of a first trajectory length corresponding to the traversed trajectory point to a length of a trajectory in which the traversed trajectory point is located; the first trajectory length includes the number of trajectory points from the traversed trajectory point to a previous trajectory point in the trajectory in which the traversed trajectory point is located, and the corresponding division action is the first division action; the length of the trajectory in which the traversed trajectory point is located includes the number of trajectory points included in the trajectory in which the traversed trajectory point is located; the second ratio may include a ratio of a second trajectory length corresponding to the traversed trajectory point to a length of the trajectory in which the traversed trajectory point is located; the second trajectory length includes the number of trajectory points from the traversed trajectory point to the last trajectory point in the trajectory in which the traversed trajectory point is located.

[0094] For example, the environment state state of the reinforcement learning can be represented as: state = (OD s , OD n , OD b , L f , Lb ) ; Among them, OD b It can represent the benchmark value of the cluster evaluation index, that is, the preset value of the cluster evaluation index; L f It can represent the first ratio corresponding to the traversed trajectory point, that is, the ratio of the number of trajectory points between the previous trajectory point on the trajectory where the traversed trajectory point is located and the traversed trajectory point to the number of trajectory points included in the trajectory where the traversed trajectory point is located; L b It can represent the second ratio corresponding to the traversed trajectory point, that is, the ratio of the number of trajectory points between the traversed trajectory point and the last trajectory point included in the trajectory where the traversed trajectory point is located to the number of trajectory points included in the trajectory where the traversed trajectory point is located.

[0095] Continue to combine Figure 3 , an example is given to illustrate the specific implementation method of determining the first ratio and the second ratio corresponding to the traversed trajectory point. Assuming that the trajectory point p4 in the trajectory T1 is traversed, then L f It can be expressed as the ratio of the first track length 1 of the sub-track sub_T1(3,4) to the track length n of the track T1, that is, L f =1 / n; L b It can be expressed as the ratio of the remaining second track length (n-3) to the track length n of track T1, that is, L b =(n-3) / n.

[0096] It should be noted that, in the embodiment shown above, by introducing more state parameters for the environment state of the reinforcement learning, the real environment can be better simulated, thereby helping the intelligent agent learn the optimal decision-making action that is closer to the real environment.

[0097] When the environment state of the reinforcement learning includes the third value and the fourth value of the cluster evaluation index, the intelligent agent can decide the division action corresponding to the traversed trajectory point based on the impact of different division actions on the value of the cluster evaluation index; when the environment state of the reinforcement learning also includes the first ratio corresponding to the traversed trajectory point, the intelligent agent can decide the division action corresponding to the traversed trajectory point based on the trajectory length of the new sub-trajectory obtained by division; when the environment state of the reinforcement learning also includes the second ratio corresponding to the traversed trajectory point, the intelligent agent can decide the division action corresponding to the traversed trajectory point based on the remaining length of the trajectory where the traversed trajectory point is located after division; when the environment state of the reinforcement learning also includes the baseline value of the cluster evaluation index, the training convergence can be accelerated.

[0098] In another embodiment shown, reinforcement learning can be implemented based on a DQN (Deep Q-learning Network) algorithm. That is, a neural network can be used to predict the function value of a value function corresponding to different decision actions, and then the action to be taken is determined based on the function value of the value function. In step 204, reinforcement learning is performed on the at least one trajectory, and each trajectory point included in each trajectory is traversed to determine the partitioning action corresponding to each trajectory point. Specifically, the method may include: traversing each trajectory point included in each trajectory; using the environmental state corresponding to the traversed trajectory point as input data and inputting it into a trained reinforcement learning-based decision model to obtain a first decision value corresponding to the first partitioning action and a second decision value corresponding to the second partitioning action output by the decision model; if the first decision value is greater than the second decision value, determining the partitioning action corresponding to the traversed trajectory point as the first partitioning action; if the first decision value is not greater than the second decision value, determining the partitioning action corresponding to the traversed trajectory point as the second partitioning action.

[0099] The first decision value can be used to represent the maximum value of the cumulative reward corresponding to the first division action under the environmental state corresponding to the traversed trajectory point; the second decision value can be used to represent the maximum value of the cumulative reward corresponding to the second division action under the environmental state corresponding to the traversed trajectory point. In other words, the first decision value can be used to represent the maximum value of the cumulative reward of the reinforcement learning when the decision action of the reinforcement learning adopts the first division action under the environmental state corresponding to the traversed trajectory point; the second decision value can be used to represent the maximum value of the cumulative reward of the reinforcement learning when the decision action of the reinforcement learning adopts the second division action under the environmental state corresponding to the traversed trajectory point.

[0100] For example, a value function Q(s, a) can be defined, which can be used to represent the maximum value of the cumulative reward that the agent will obtain after taking decision action a in the environmental state s. Assuming that the trajectory point p4 in trajectory T1 is traversed, the environmental state state corresponding to trajectory point p4 can be input into the trained decision model to obtain the first decision value Qvalue1 corresponding to the first division action and the second decision value Qvalue2 corresponding to the second division action output by the decision model; further, since the learning goal of reinforcement learning is to maximize the value of the cumulative reward REWARD, if Qvalue1>Qvalue2, the division action corresponding to trajectory point p4 can be determined to be the first division action, that is, the trajectory T1 is divided at the position of trajectory point p4; otherwise, the division action corresponding to trajectory point p4 can be determined to be the second division action, that is, the trajectory T1 is not divided at the position of trajectory point p4.

[0101] It should be noted that, in the above-described embodiments, in addition to the DQN algorithm, a Q-learning algorithm or the like may also be used to implement reinforcement learning. This is not limited in this specification and will not be described in detail here.

[0102] The decision model may specifically include an MDP (Markov Decision Process) model based on a neural network. That is, a decision model may be constructed based on a neural network to perform reinforcement learning. Specifically, the decision model may be modeled as an MDP model.

[0103] In another embodiment shown, in step 204, clustering the sub-trajectories obtained by the division to obtain k sub-trajectory sets may specifically include: calculating the distance between each sub-trajectory obtained by the division and the k cluster centers; and dividing each sub-trajectory into the sub-trajectory set corresponding to the cluster center closest to it.

[0104] For example, after dividing the trajectory of each trajectory point by using the division action corresponding to each trajectory point, the sub-trajectories obtained by division can be calculated and the k cluster centers c1, c2, ..., c k Furthermore, each sub-trajectory can be divided into the sub-trajectory set corresponding to the cluster center closest to it.

[0105] Step 206: Obtain k sub-trajectory sets obtained based on reinforcement learning as sub-trajectory clustering results for the trajectory set.

[0106] For example, in the trajectory set to be divided {T1, T2, ..., TM After reinforcement learning is performed on the M trajectories included in}, k sub-trajectory sets C1, C2, ..., C k , as a set of trajectories {T1, T2, ..., T M It should be noted that, according to the sub-trajectory clustering result for the trajectory set, the trajectory division result for the trajectory set can be determined, that is, at least one sub-trajectory included in the k sub-trajectory sets obtained based on reinforcement learning.

[0107] In another embodiment, reinforcement learning is iteratively performed on the trajectory set until the reinforcement learning iteration stop condition is met. In step 206, obtaining k sub-trajectory sets obtained through reinforcement learning as the sub-trajectory clustering result for the trajectory set may specifically include: updating the cluster center of each sub-trajectory set based on the k sub-trajectory sets obtained through clustering; if the distance between the updated cluster center of each sub-trajectory set and the pre-update cluster center does not exceed a preset threshold, stopping the reinforcement learning iteration process and obtaining the k sub-trajectory sets obtained through clustering as the sub-trajectory clustering result for the trajectory set.

[0108] For example, see Figure 4 , Figure 4 FIG is a flowchart of another sub-trajectory clustering method based on reinforcement learning, which is shown in an exemplary embodiment. Figure 4 As shown, after obtaining the trajectory set to be divided {T1, T2, ..., T M After that, reinforcement learning can be performed on the M trajectories included in the trajectory set, and each trajectory point included in each trajectory is traversed to decide the division action corresponding to each trajectory point, and the division action corresponding to each trajectory point is used to divide the trajectory where each trajectory point is located to obtain at least one sub-trajectory, and clustering is performed on each sub-trajectory obtained by division to obtain k sub-trajectory sets; based on the k sub-trajectory sets obtained by clustering, the cluster center of each sub-trajectory set is updated; if the distance between the updated cluster center of each sub-trajectory set and the cluster center before the update does not exceed the preset threshold, the iterative process of the reinforcement learning is stopped, and the k sub-trajectory sets obtained by clustering are obtained as the sub-trajectory clustering result for the trajectory set; if the distance between the updated cluster center of each sub-trajectory set and the cluster center before the update exceeds the preset threshold, the iterative process of the reinforcement learning is continued.

[0109] It should be noted that in the implementation shown above, the distance between the updated cluster center of each sub-trajectory set and the cluster center before the update does not exceed the preset threshold, indicating that the k sub-trajectory sets obtained in each round of iterative process have tended to be stable, and it can be considered that the intelligent agent has completed reinforcement learning for the trajectory set.

[0110] In one or more of the embodiments described above, when calculating the distance between the partitioned subtrajectories and the cluster centers, and updating the cluster centers of each subtrajectory set, the similarity between two trajectory point sequences can be measured by calculating the distance between them. Regarding the specific implementation of calculating the distance between two trajectory point sequences, discrete Frechet distance, DTW, LCSS, EDR, etc. can be used. Those skilled in the art can flexibly select different methods based on their needs, and this specification does not specifically limit this. The following briefly describes two distance calculation methods.

[0111] In a possible example, for the line segment between the start point and the end point of each of the two trajectories, the parallel distance d between the two line segments can be calculated. || , vertical distance d ⊥ , and angular distance d θ The weighted sum D of is taken as the distance between the two trajectories.

[0112] For example, see Figure 5 , Figure 5 FIG. 1 is a schematic diagram showing distance calculation according to an exemplary embodiment. Figure 5 As shown, a trajectory T i The starting point is the trajectory point s i , trajectory T i The end point is the trajectory point e i ; Another trajectory T j The starting point is the trajectory point s j , trajectory T j The end point is the trajectory point e j ; Further, the line segment s can be calculated by the following formula i e i With line segment s j e j The parallel distance d || , vertical distance d ⊥ , and angular distance d θ :

[0113] d || =MIN(l ||1 , l ||2 )

[0114]

[0115] d θ =||T j ||×sin(θ)

[0116] Among them, l ||1 Represents the trajectory point s i With trajectory point s j The parallel distance between ||2 Represents the trajectory point e i With trajectory point e j The parallel distance between ||1 , l ||2 ) means taking l ||1 With l ||2 The smaller value between ⊥1 Represents the trajectory point s i With trajectory point s j The vertical distance between ⊥2 Represents the trajectory point e i With trajectory point e j The vertical distance between ||T j || represents the trajectory T j The length of the trajectory T i With trajectory T j The angle between them; further, if the parallel distance d || , vertical distance d ⊥ , and angular distance d θ The corresponding weight coefficients are α1, α2, and α3 respectively, then the trajectory T i With trajectory T j The distance D between them can be represented as: D = α1 × d || +α2×d ⊥ +α3×d θ .

[0117] In another possible example, the discrete Frechet distance between two trajectories can be calculated.

[0118] For example, the discrete Frechet distance d between trajectory Q and trajectory T DFD (Q, T) can be characterized as:

[0119]

[0120] Where d(p,q) represents the Euclidean distance between trajectory point p and trajectory point q; Q h represents the sub-trajectory starting from the second trajectory point in trajectory Q; T h Represents the sub-trajectory starting from the second trajectory point in trajectory R.

[0121] Through the above technical solutions, it can be seen that, on the one hand, by treating the trajectory division process as a sequential decision-making process, reinforcement learning can be performed on at least one trajectory included in the trajectory set (that is, the trajectory point sequence), traversing each trajectory point included in each trajectory to decide the division action corresponding to each trajectory point. In this way, it is possible to adapt to the trajectory division requirements of the sub-trajectory clustering problem in different application scenarios based on the intrinsic characteristics of the different trajectories to be divided, avoiding the manual formulation of different trajectory division rules, and can be completely data-driven, making it more universal. On the other hand, solving the sub-trajectory clustering problem based on reinforcement learning can use the sub-trajectory clustering results to guide the trajectory division process, thereby learning the trajectory division decision that can produce the optimal sub-trajectory clustering results, and ultimately obtaining a better sub-trajectory clustering result.

[0122] Corresponding to the above embodiment of the sub-trajectory clustering method based on reinforcement learning, this specification also provides an embodiment of a sub-trajectory clustering system based on reinforcement learning, and an embodiment of a sub-trajectory clustering device based on reinforcement learning.

[0123] In this specification, the sub-trajectory clustering system based on reinforcement learning can execute steps 202 to 206 to implement the technical solution of this specification.

[0124] See Figure 6 , Figure 6 This is a hardware structure diagram of an electronic device in which a sub-trajectory clustering device based on reinforcement learning is located, shown as an exemplary embodiment. At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610, and of course may also include hardware required for other services. One or more embodiments of this specification can be implemented based on software, such as the processor 602 reading the corresponding computer program from the non-volatile memory 610 into the memory 608 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0125] See Figure 7 , Figure 7 FIG is a block diagram of a sub-trajectory clustering device based on reinforcement learning, which is shown in an exemplary embodiment. The sub-trajectory clustering device based on reinforcement learning can be applied to Figure 6 In the electronic device shown in the figure, the technical solution of this specification is implemented. The sub-trajectory clustering device based on reinforcement learning may include:

[0126] An acquisition unit 702 is configured to acquire a trajectory set to be divided; wherein the trajectory set includes at least one trajectory; the at least one trajectory includes k trajectories used as initial cluster centers, where k is a positive integer; and the trajectory is a trajectory point sequence including at least one trajectory point.

[0127] a reinforcement learning unit 704 configured to perform reinforcement learning on the at least one trajectory, traverse each trajectory point included in each trajectory to determine a partitioning action corresponding to each trajectory point, and use the partitioning action corresponding to each trajectory point to partition the trajectory containing each trajectory point to obtain at least one sub-trajectory, and cluster each of the partitioned sub-trajectories to obtain k sub-trajectory sets, such that the values ​​of clustering evaluation indicators of the k sub-trajectory sets obtained by clustering meet expected values;

[0128] The obtaining unit 706 is configured to obtain k sub-trajectory sets obtained based on reinforcement learning as sub-trajectory clustering results for the trajectory set.

[0129] In this embodiment, the decision-making action of the reinforcement learning includes: a division action of whether to divide the trajectory where the traversed trajectory point is located; wherein the division action includes a first division action of dividing the trajectory where the trajectory point is located, and a second division action of not dividing the trajectory where the trajectory point is located;

[0130] The reward corresponding to the reinforcement learning decision action includes: the value of a clustering evaluation index for the k sub-trajectories obtained by clustering the sub-trajectories obtained by dividing the trajectory containing the traversed trajectory point using the partitioning action corresponding to the traversed trajectory point; wherein the clustering evaluation index includes the average distance between each sub-trajectory and the cluster center of the sub-trajectory set to which it belongs.

[0131] In this embodiment, the values ​​of the clustering evaluation index of the k sub-trajectory sets meet the expected values, including: the average value of the distance from each sub-trajectory to the cluster center of the sub-trajectory set to which it belongs is the smallest;

[0132] The clustering evaluation index is characterized by the following formula:

[0133]

[0134]

[0135] Among them, OD represents the clustering evaluation index of the k sub-trajectory set; m represents the total number of sub-trajectories included in the k sub-trajectory set; m i Represents the i-th sub-trajectory set C i The number of sub-trajectories included in OD iRepresents the i-th sub-trajectory set C i Clustering evaluation index; x represents the set of sub-trajectories belonging to the i-th sub-trajectory set C i The sub-trajectory included in c i Represents the i-th sub-trajectory set C i The cluster center of dist(x,c i ) represents the sub-trajectory x to the sub-trajectory set C to which it belongs i The cluster center c i distance.

[0136] In this embodiment, the cumulative reward of the reinforcement learning includes: the sum of the instantaneous rewards corresponding to each traversed trajectory point;

[0137] The instant reward corresponding to the traversed trajectory point includes: the difference between the first value and the second value of the clustering evaluation index corresponding to the traversed trajectory point; wherein, the first value includes the value of the clustering evaluation index before the division action corresponding to the traversed trajectory point is adopted to divide the trajectory where the trajectory point is located; the second value includes the value of the clustering evaluation index after the division action corresponding to the traversed trajectory point is adopted to divide the trajectory where the trajectory point is located, and clustering the sub-trajectories obtained by the division.

[0138] In this embodiment, the environmental state related to the decision-making action of the reinforcement learning includes: a third value and a fourth value of the clustering evaluation index corresponding to the traversed trajectory point; wherein the third value includes the value of the clustering evaluation index after the traversed trajectory point is divided by the first division action, and each sub-trajectory obtained by the division is clustered; the fourth value includes the value of the clustering evaluation index after the traversed trajectory point is not divided by the second division action, and each sub-trajectory obtained by the division is clustered.

[0139] In this embodiment, the environmental state related to the decision-making action of the reinforcement learning further includes: one or more of the benchmark value of the cluster evaluation index, the first ratio corresponding to the traversed trajectory point, and the second ratio corresponding to the traversed trajectory point;

[0140] Wherein, the benchmark value includes the preset value of the cluster evaluation index;

[0141] The first ratio includes a ratio of a first trajectory length corresponding to the traversed trajectory point to a length of a trajectory in which the traversed trajectory point is located; the first trajectory length includes the number of trajectory points between the traversed trajectory point and a previous trajectory point in the trajectory in which the traversed trajectory point is located, the corresponding division action of which is the first division action; the length of the trajectory in which the traversed trajectory point is located includes the number of trajectory points in the trajectory in which the traversed trajectory point is located;

[0142] The second ratio includes the ratio of the second trajectory length corresponding to the traversed trajectory point to the length of the trajectory where the traversed trajectory point is located; the second trajectory length includes the number of trajectory points between the traversed trajectory point and the last trajectory point included in the trajectory where the traversed trajectory point is located.

[0143] In this embodiment, the reinforcement learning unit 704 is specifically configured to:

[0144] Traversing each trajectory point included in each trajectory;

[0145] The environmental state corresponding to the traversed trajectory point is used as input data and input into a trained reinforcement learning-based decision model to obtain a first decision value corresponding to the first division action and a second decision value corresponding to the second division action output by the decision model; wherein the first decision value is used to represent the maximum value of the cumulative reward corresponding to the first division action under the environmental state corresponding to the traversed trajectory point; and the second decision value is used to represent the maximum value of the cumulative reward corresponding to the second division action under the environmental state corresponding to the traversed trajectory point;

[0146] If the first decision value is greater than the second decision value, determining the division action corresponding to the traversed trajectory point as the first division action;

[0147] If the first decision value is not greater than the second decision value, the division action corresponding to the traversed trajectory point is determined to be the second division action.

[0148] In this embodiment, the decision model includes: an MDP model based on a neural network.

[0149] In this embodiment, the obtaining unit 706 is configured to:

[0150] Based on the k sub-trajectory sets obtained by clustering, update the cluster center of each sub-trajectory set;

[0151] If the distance between the updated cluster center and the pre-updated cluster center of each sub-trajectory set does not exceed a preset threshold, the iterative process of the reinforcement learning is stopped, and the k sub-trajectory sets obtained by clustering are obtained as the sub-trajectory clustering result for the trajectory set.

[0152] In this embodiment, the reinforcement learning unit 704 is specifically configured to:

[0153] Calculate the distance between each sub-trajectory obtained by division and the k cluster centers;

[0154] Each sub-trajectory is divided into the sub-trajectory set corresponding to the cluster center closest to it.

[0155] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0156] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this specification. Ordinary technicians in this field can understand and implement it without paying any creative work.

[0157] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0158] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0159] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0160] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0161] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0162] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0163] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0164] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0165] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A sub-trajectory clustering method based on reinforcement learning, the method comprising: Obtain a trajectory set to be divided; wherein the trajectory set includes at least one trajectory; the at least one trajectory includes k trajectories used as initial cluster centers, where k is a positive integer; the trajectory is a trajectory point sequence including at least one trajectory point of a moving object; the moving object includes a pedestrian, a vehicle, or an aircraft; the trajectory point sequence includes Track points , Indicates the trajectory Track points, is a positive integer, the trajectory point for ,in, 、 Represents trajectory points The longitude and latitude coordinates of Represents trajectory points The sampling time of performing reinforcement learning on the at least one trajectory, traversing each trajectory point included in each trajectory to determine a partitioning action corresponding to each trajectory point, partitioning the trajectory containing each trajectory point using the partitioning action corresponding to each trajectory point to obtain at least one sub-trajectory, and clustering each of the sub-trajectories obtained by the partitioning to obtain k sub-trajectory sets, such that a clustering evaluation index of each of the k sub-trajectory sets obtained by clustering meets an expected value; Obtaining k sub-trajectory sets obtained based on reinforcement learning as sub-trajectory clustering results for the trajectory set; The decision-making action of the reinforcement learning includes: a division action of whether to divide the trajectory where the traversed trajectory point is located; wherein the division action includes a first division action of dividing the trajectory where the trajectory point is located, and a second division action of not dividing the trajectory where the trajectory point is located; The reward corresponding to the reinforcement learning decision action includes: the value of a clustering evaluation index for the k sub-trajectories obtained by clustering the sub-trajectories obtained by dividing the trajectory containing the traversed trajectory point using the partitioning action corresponding to the trajectory point; wherein the clustering evaluation index includes the average distance between each sub-trajectory and the cluster center of the sub-trajectory set to which it belongs; The reinforcement learning is performed on the at least one trajectory, and each trajectory point included in each trajectory is traversed to determine a division action corresponding to each trajectory point, including: traversing each trajectory point included in each trajectory; using the environmental state corresponding to the traversed trajectory point as input data, and inputting the trained reinforcement learning-based decision model to obtain a first decision value corresponding to the first division action and a second decision value corresponding to the second division action output by the decision model; wherein the first decision value is used to represent the maximum value of the cumulative reward corresponding to the first division action under the environmental state corresponding to the traversed trajectory point; and the second decision value is used to represent the maximum value of the cumulative reward corresponding to the second division action under the environmental state corresponding to the traversed trajectory point; if the first decision value is greater than the second decision value, determining that the division action corresponding to the traversed trajectory point is the first division action; if the first decision value is not greater than the second decision value, determining that the division action corresponding to the traversed trajectory point is the second division action; The environment state includes information related to the decision-making action of the reinforcement learning, and the cumulative reward of the reinforcement learning includes the sum of the instantaneous rewards corresponding to each traversed trajectory point.

2. The method according to claim 1, wherein the values ​​of the clustering evaluation index of the k sub-trajectory sets meet the expected values, comprising: The average value of the distance between each sub-trajectory and the cluster center of the sub-trajectory set to which it belongs is the smallest; The clustering evaluation index is characterized by the following formula: in, express Clustering evaluation index of sub-trajectory sets; express The total number of sub-trajectories included in the sub-trajectory set; Indicates the Sub-trajectory set The number of subtrajectories included in ; Indicates the Sub-trajectory set Clustering evaluation index; Indicates that it belongs to Sub-trajectory set The subtrajectories included in ; Indicates the Sub-trajectory set The cluster center of Represents a sub-track To the sub-trajectory set to which it belongs The cluster center distance.

3. The method according to claim 1, The cumulative rewards of the reinforcement learning include: The sum of the immediate rewards corresponding to each traversed trajectory point; The instant reward corresponding to the traversed trajectory point includes: the difference between the first value and the second value of the clustering evaluation index corresponding to the traversed trajectory point; wherein, the first value includes the value of the clustering evaluation index before the division action corresponding to the traversed trajectory point is adopted to divide the trajectory where the trajectory point is located; the second value includes the value of the clustering evaluation index after the division action corresponding to the traversed trajectory point is adopted to divide the trajectory where the trajectory point is located, and clustering the sub-trajectories obtained by the division.

4. The method according to claim 1, The state of the environment related to the decision action of the reinforcement learning includes: The third and fourth values ​​of the clustering evaluation index corresponding to the traversed trajectory point; wherein the third value includes the value of the clustering evaluation index after the trajectory where the traversed trajectory point is located is divided by the first division action, and each sub-trajectory obtained by the division is clustered; the fourth value includes the value of the clustering evaluation index after the trajectory where the traversed trajectory point is located is not divided by the second division action, and each sub-trajectory obtained by the division is clustered.

5. The method according to claim 4, The environment state related to the decision action of the reinforcement learning also includes: One or more of a benchmark value of the cluster evaluation index, a first ratio corresponding to the traversed trajectory point, and a second ratio corresponding to the traversed trajectory point; Wherein, the benchmark value includes the preset value of the cluster evaluation index; The first ratio includes a ratio of a first trajectory length corresponding to the traversed trajectory point to a length of a trajectory in which the traversed trajectory point is located; the first trajectory length includes the number of trajectory points between the traversed trajectory point and a previous trajectory point in the trajectory in which the traversed trajectory point is located, the corresponding division action of which is the first division action; the length of the trajectory in which the traversed trajectory point is located includes the number of trajectory points in the trajectory in which the traversed trajectory point is located; The second ratio includes the ratio of the second trajectory length corresponding to the traversed trajectory point to the length of the trajectory where the traversed trajectory point is located; the second trajectory length includes the number of trajectory points between the traversed trajectory point and the last trajectory point included in the trajectory where the traversed trajectory point is located.

6. The method according to claim 1, wherein the decision model comprises: MDP model based on neural network.

7. The method according to claim 1, wherein obtaining k sub-trajectory sets obtained based on reinforcement learning as sub-trajectory clustering results for the trajectory set comprises: Based on the k sub-trajectory sets obtained by clustering, update the cluster center of each sub-trajectory set; If the distance between the updated cluster center and the pre-updated cluster center of each sub-trajectory set does not exceed a preset threshold, the iterative process of the reinforcement learning is stopped, and the k sub-trajectory sets obtained by clustering are obtained as the sub-trajectory clustering result for the trajectory set.

8. The method according to claim 1, wherein clustering the sub-trajectories obtained by the division to obtain k sub-trajectory sets comprises: Calculate the distance between each sub-trajectory obtained by division and the k cluster centers; Each sub-trajectory is divided into the sub-trajectory set corresponding to the cluster center closest to it.

9. A sub-trajectory clustering system based on reinforcement learning, wherein the sub-trajectory clustering system is used to implement the method according to any one of claims 1 to 8.

10. An electronic device comprising a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the method according to any one of claims 1 to 8 by calling the machine-readable instructions.

11. A machine-readable storage medium storing machine-readable instructions, wherein the machine-readable instructions, when called and executed by a processor, implement the method according to any one of claims 1 to 8.

Images