Vehicle trajectory recovery method, device, apparatus and storage medium

Abstract

The embodiment of the present invention provides a vehicle trajectory recovery method, equipment, device and storage medium, the method comprising: obtaining visual features of multiple vehicles obtained based on multiple traffic cameras; obtaining multimodal features of each vehicle based on the visual features of multiple vehicles and the spatiotemporal features of the vehicle trajectory constraints; obtaining data of multiple trajectory sampling points of each vehicle based on the multimodal features of each vehicle; obtaining trajectory recovery results of each vehicle based on a path probability model and the data of the trajectory sampling points of each vehicle; the path probability model is used to determine the path of each vehicle with the maximum probability between adjacent trajectory sampling points. The method of the embodiment of the present invention, based on the visual features of the vehicle, fully considers the spatiotemporal rationality constraints of the trajectory and accurately realizes the recovery of the vehicle trajectory.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic information processing, and in particular to a vehicle trajectory recovery method, equipment, device and storage medium. Background Art

[0002] Vehicle trajectory data supports traffic perception, management, and decision-making by traffic management departments. It is also an important basis for downstream applications such as traffic forecasting, congestion analysis, logistics distribution, and taxi dispatch.

[0003] In the related art, only the trajectory recovery can be performed for a specific target vehicle to obtain the vehicle trajectory data, but the trajectories of multiple vehicles cannot be recovered. Therefore, how to recover the trajectories of multiple vehicles is an issue that those skilled in the art need to pay attention to. Summary of the invention

[0004] In view of the problems in the prior art, embodiments of the present invention provide a vehicle trajectory recovery method, device, apparatus and storage medium.

[0005] Specifically, the embodiment of the present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a vehicle trajectory recovery method, comprising:

[0007] Obtain visual features of multiple vehicles based on multiple traffic cameras;

[0008] Acquire multimodal features of each of the vehicles according to visual features of the multiple vehicles and spatiotemporal features of trajectory constraints on the vehicles;

[0009] Acquiring data of a plurality of trajectory sampling points of each of the vehicles according to the multimodal characteristics of each of the vehicles;

[0010] According to the path probability model and the data of the trajectory sampling points of each of the vehicles, the trajectory recovery results of each of the vehicles are obtained; the path probability model is used to determine the path with the maximum probability of each vehicle between adjacent trajectory sampling points.

[0011] Further, the visual features of the vehicle include at least one of the following: an overall visual feature of the vehicle and a visual feature of a license plate of the vehicle; and according to the visual features of the multiple vehicles and the spatiotemporal features of the vehicles, obtaining multimodal features of each of the vehicles includes:

[0012] Determining a multimodal similarity between visual features of each of the vehicles based on the visual features of the multiple vehicles and the spatiotemporal features of the vehicles;

[0013] The multimodal features of each of the vehicles are acquired according to the multimodal similarity between the visual features of each of the vehicles and a first threshold.

[0014] Furthermore, the path probability model is determined based on a trained path prior probability model and a trained path likelihood probability model; the path prior probability model is trained based on first training data; the first training data includes: information on each path between the first trajectory sampling point and the second trajectory sampling point passed by each vehicle in the historical trajectory; the path likelihood probability model is trained based on second training data; the second training data includes: vehicle speed information on each path between the first trajectory sampling point and the second trajectory sampling point passed by each vehicle in each time period in the historical trajectory.

[0015] Furthermore, the path prior probability model is determined using the following formula (1):

[0016]

[0017] Among them, r s represents the first trajectory sampling point; r e represents the second trajectory sampling point; p represents any path between the first trajectory sampling point and the second trajectory sampling point; the path p is composed of the road segment s i Composition, p = {s1, ..., s n}; Pr(p|r s , r e ) represents the prior probability of the vehicle passing through path p; Pr(s i+1 |s i , r e ) indicates that the sampling point of the second trajectory is r e , the current section is s i When , the next section is s i+1 The prior probability of s , r e ) indicates that the sampling point of the second trajectory is r e , the probability that the first road segment is s1.

[0018] Furthermore, the path likelihood probability model is determined using the following formula (2):

[0019]

[0020] Where p represents any path between the first trajectory sampling point and the second trajectory sampling point; r s represents the first trajectory sampling point; r e represents the sampling point interval of the second trajectory; Indicates that at time t eThe total expected travel time of the vehicle passing through each path between the first trajectory sampling point and the second trajectory sampling point under the historical trajectory; σ represents the standard deviation of the travel time of the vehicle passing through each path between the first trajectory sampling point and the second trajectory sampling point in the historical trajectory; Pr(Δt|p, t e ) indicates that at time t e Under this condition, the likelihood probability that the vehicle passes through the path p in a travel time of Δt.

[0021] Furthermore, after obtaining the trajectory recovery result of each vehicle according to the path probability model and the multimodal characteristics of each vehicle, the method further includes:

[0022] According to the trajectory recovery results of each of the vehicles, noise sampling points and missing sampling points in the trajectory sampling points corresponding to the multimodal features of each of the vehicles are obtained;

[0023] Updating the spatiotemporal features according to the noise sampling points and the missing sampling points;

[0024] According to the path probability model and the data of the updated trajectory sampling points of each of the vehicles, the target trajectory recovery result of each of the vehicles is obtained; the data of the updated trajectory sampling points of each of the vehicles is determined based on the updated spatiotemporal features.

[0025] In a second aspect, an embodiment of the present invention further provides a vehicle trajectory recovery device, comprising:

[0026] An acquisition module is used to acquire visual features of multiple vehicles obtained based on multiple traffic cameras; acquire multimodal features of each of the vehicles based on the visual features of the multiple vehicles and the spatiotemporal features of the trajectory constraints on the vehicles; and acquire data of multiple trajectory sampling points of each of the vehicles based on the multimodal features of each of the vehicles;

[0027] The recovery module is used to obtain the trajectory recovery result of each vehicle according to the path probability model and the data of the trajectory sampling points of each vehicle; the path probability model is used to determine the path with the maximum probability of each vehicle between adjacent trajectory sampling points.

[0028] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the vehicle trajectory recovery method as described in the first aspect is implemented.

[0029] In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the vehicle trajectory recovery method as described in the first aspect is implemented.

[0030] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the vehicle trajectory recovery method as described in the first aspect.

[0031] The vehicle trajectory recovery method, equipment, device and storage medium provided by the embodiments of the present invention obtain the multimodal features of each vehicle based on the visual features of multiple vehicles and the spatiotemporal features of the trajectory constraints of the vehicles, that is, on the basis of the visual features of the vehicles, the spatiotemporal rationality constraints of the trajectories are fully considered, thereby improving the accuracy of vehicle clustering, and making the multimodal features of each vehicle obtained more accurate and reasonable, thereby improving the accuracy of path recovery; further, the trajectory sampling points of each vehicle are obtained based on the multimodal features of the vehicles, that is, based on the correspondence between the multimodal feature data and the camera position, the position and time information of each vehicle passing by can be accurately determined, and can be used as the trajectory sampling points of each vehicle; finally, based on the path probability model and the data of the trajectory sampling points of each vehicle, the trajectory recovery results of each vehicle can be accurately and effectively obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 This is one of the flow charts of the vehicle trajectory recovery method provided by an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of a process for obtaining multimodal features of a vehicle provided by an embodiment of the present invention;

[0035] Figure 3 This is a second flow chart of the vehicle trajectory recovery method provided by an embodiment of the present invention;

[0036] Figure 4 is a schematic diagram of a process for obtaining noise sampling points and missing sampling points provided by an embodiment of the present invention;

[0037] Figure 5 This is a third flow chart of the vehicle trajectory recovery method provided by an embodiment of the present invention;

[0038] Figure 6 This is a fourth flow chart of the vehicle trajectory recovery method provided by an embodiment of the present invention;

[0039] Figure 7is a schematic diagram of a vehicle trajectory recovery result provided by an embodiment of the present invention;

[0040] Figure 8 is a structural schematic diagram of a vehicle trajectory recovery device provided by an embodiment of the present invention;

[0041] Fig. 9 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The method of the embodiment of the present invention can be applied to traffic information processing scenarios to achieve vehicle trajectory recovery.

[0044] In the related art, only the trajectory recovery can be performed for a specific target vehicle to obtain the vehicle trajectory data, but the trajectories of multiple vehicles cannot be recovered. Therefore, how to recover the trajectories of multiple vehicles is an issue that those skilled in the art need to pay attention to.

[0045] The vehicle trajectory recovery method of the embodiment of the present invention obtains the multimodal features of each vehicle based on the visual features of multiple vehicles and the spatiotemporal features of the vehicle trajectory constraints, that is, on the basis of the visual features of the vehicles, the spatiotemporal rationality constraints of the trajectories are fully considered, thereby improving the accuracy of vehicle clustering, and making the multimodal features of each vehicle obtained more accurate and reasonable, thereby improving the accuracy of path recovery; further, the trajectory sampling points of each vehicle are obtained based on the multimodal features of the vehicles, that is, based on the correspondence between the multimodal feature data and the camera position, the position and time information of each vehicle passing by can be accurately determined, and can be used as the trajectory sampling points of each vehicle; finally, based on the path probability model and the data of the trajectory sampling points of each vehicle, the trajectory recovery results of each vehicle can be accurately and effectively obtained.

[0046] Combine the following Figure 1-Figure 9 The technical solution of the present invention is described in detail with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0047] Figure 1 FIG. 1 is a flow chart of an embodiment of a vehicle trajectory recovery method provided by an embodiment of the present invention. Figure 1As shown, the method provided in this embodiment includes:

[0048] Step 101: Acquire visual features of multiple vehicles based on multiple traffic cameras;

[0049] Specifically, vehicle trajectory data supports traffic perception, management, and decision-making, and is also an important basis for downstream applications such as traffic forecasting, congestion analysis, logistics distribution, and taxi dispatch. The high-sampling rate vehicle trajectory data that can be directly obtained currently comes from a small number of taxis or vehicles with navigation app services turned on, and it is difficult to reliably support the estimation of traffic conditions on the entire network and downstream applications. In addition, the existing vehicle trajectory recovery system restores the complete spatiotemporal trajectory of the vehicle on the road network based on the low-sampling and interference information of the given vehicle trajectory. It only infers the unobserved movement path based on the low-sampling rate trajectory of the determined vehicle, and cannot recover the trajectory of each vehicle from a large amount of vehicle data without identity tags.

[0050] In an embodiment of the present application, by acquiring visual features of multiple vehicles based on multiple traffic cameras, that is, by obtaining visual features of multiple vehicles from video data of traffic cameras widely deployed in urban road networks, rich and reliable visual feature data of multiple vehicles are obtained.

[0051] For example, for the acquired traffic camera video data, through frame extraction, vehicle target detection and visual feature extraction, the event of a car passing through the camera is converted into a representative image and corresponding visual features.

[0052] Step 102: acquiring multimodal features of each vehicle based on visual features of the multiple vehicles and spatiotemporal features of trajectory constraints on the vehicles;

[0053] Specifically, the prior art only retrieves all the data of a specific target vehicle from certain data, that is, only identifies the vehicle by similarity of visual features, and cannot accurately and effectively solve the clustering problem of identifying and distinguishing each vehicle from a large number of vehicles.

[0054] In the embodiment of the present application, the multimodal features of each vehicle are obtained based on the visual features of multiple vehicles and the spatiotemporal features of the vehicle's trajectory constraints, and the information of which cameras each vehicle has passed through, as well as the corresponding position information and the corresponding time information are obtained. Among them, the spatiotemporal features can be preset spatiotemporal features of the vehicle's trajectory constraints, or they can be spatiotemporal features of the vehicle's trajectory constraints obtained based on noise sampling points in historical trajectories or restored vehicle trajectories. By fully considering the spatiotemporal rationality constraints of the trajectory, this spatiotemporal constraint information can be directly used in the clustering process, which improves the accuracy and rationality of clustering, and also improves the accuracy of path recovery.

[0055] For example, the visual features of a vehicle include the color of the vehicle. After clustering the vehicles according to their colors, we obtain the visual features of three vehicles, red, yellow and blue. Among them, the red vehicle corresponds to images of five visual features, that is, the images corresponding to the five visual features represent the images of the visual features of the red vehicle when it passes through the 1st to 5th cameras respectively; and the preset spatiotemporal features include constraints on the path corresponding to the 5th camera (for example, in the preset spatiotemporal features, the road corresponding to the location of the 5th camera is a one-way street and can only be driven from east to west, while at the trajectory sampling points corresponding to the above 5 visual features, the red vehicle happens to be driving from west to east). Then, based on the visual features of the vehicle and the spatiotemporal features of the trajectory constraints on the vehicle, after verification, the multimodal features of the red vehicle only include the data of the visual features corresponding to the 1st to 4th cameras, but do not include the data of the visual features corresponding to the 5th camera.

[0056] Step 103: acquiring data of multiple trajectory sampling points of each vehicle according to the multimodal characteristics of each vehicle;

[0057] Specifically, after obtaining the multimodal features of each vehicle, we can also obtain which cameras each vehicle passed by and the time each vehicle passed by the camera. Accordingly, the position and time corresponding to the multimodal features can be used as the trajectory sampling point of each vehicle.

[0058] For example, the multimodal feature data of vehicle A includes the visual feature data corresponding to the 1st to 4th cameras when vehicle A passes through. Based on the position of the camera and the corresponding visual feature image, the position and time passed by vehicle A can be determined, and it can be used as the trajectory sampling point in the trajectory to be restored corresponding to vehicle A.

[0059] Step 104: Obtain the trajectory recovery result of each vehicle according to the path probability model and the data of the trajectory sampling points of each vehicle; the path probability model is used to determine the path with the maximum probability of each vehicle between adjacent trajectory sampling points.

[0060] Specifically, the path probability model is used to determine the path with the maximum probability of each vehicle between adjacent trajectory sampling points. The input of the path probability model is the data of the trajectory sampling points of each vehicle, and the output of the path probability model is the trajectory recovery result between the trajectory sampling points. The path probability model is trained based on historical trajectory data. Accordingly, after obtaining the data of multiple trajectory sampling points of each vehicle, the data of the trajectory sampling points of each vehicle are input into the path probability model, so that the driving path of each vehicle between each trajectory sampling point can be determined, and the trajectory recovery result of each vehicle can be obtained.

[0061] For example, there are 5 paths from the first trajectory sampling point to the second trajectory sampling point. According to the path probability model, it is determined that the probability of vehicle A selecting the first path from the first trajectory sampling point to the second trajectory sampling point is 5%, the probability of selecting the second path is 10%, the probability of selecting the third path is 5%, the probability of selecting the fourth path is 10%, and the probability of selecting the fifth path is 70%. Optionally, the fifth path with the largest probability is determined as the driving path of vehicle A from the first trajectory sampling point to the second trajectory sampling point, and the trajectory recovery result of vehicle A is obtained.

[0062] The method of the above embodiment obtains the multimodal features of each vehicle based on the visual features of multiple vehicles and the spatiotemporal features of the vehicle trajectory constraints, that is, on the basis of the visual features of the vehicle, the spatiotemporal rationality constraints of the trajectory are fully considered, thereby improving the accuracy of vehicle clustering, and making the multimodal features of each vehicle obtained more accurate and reasonable, thereby improving the accuracy of path recovery; further, the trajectory sampling points of each vehicle are obtained based on the multimodal features of the vehicle, that is, based on the correspondence between the multimodal feature data and the camera position, the position and time information of each vehicle passing by can be accurately determined, and can be used as the trajectory sampling points of each vehicle; finally, based on the path probability model and the data of the trajectory sampling points of each vehicle, the trajectory recovery results of each vehicle can be accurately and effectively obtained.

[0063] Optionally, the visual features of the vehicle include at least one of the following: an overall visual feature of the vehicle and a visual feature of a license plate of the vehicle; and obtaining multimodal features of each vehicle according to the visual features of multiple vehicles and the spatiotemporal features of the vehicles includes:

[0064] Determine the multimodal similarity between the visual features of each vehicle based on the visual features of the multiple vehicles and the spatiotemporal features of the vehicles;

[0065] The multimodal features of each vehicle are obtained according to the multimodal similarity between the visual features of each vehicle and a first threshold.

[0066] Specifically, in the embodiment of the present application, the obtained vehicle visual features include not only the overall visual features of the vehicle, but also the visual features of the vehicle's license plate. Then, based on the overall visual features of the vehicle and the visual features of the vehicle's license plate, the clustering results of the vehicles can be made more accurate, and the multimodal features of each vehicle can be accurately obtained. In addition, when obtaining the multimodal features of each vehicle, not only the visual features of the vehicle are considered, but also the temporal and spatial rationality constraints of the trajectory are fully considered, so that the temporal and spatial constraint information can be directly used in the clustering process, thereby improving the accuracy and rationality of the clustering, and thus improving the accuracy of the multimodal features of each vehicle and the accuracy of the path recovery of each vehicle. Optionally, firstly based on the overall visual features of the vehicle f a, license plate visual features p , spatiotemporal characteristics d , define the multimodal similarity S between the sample points i and j corresponding to the vehicle i，j as follows:

[0067]

[0068] where f a , f p Both are 256-dimensional feature vectors, f d Initialize to f a , the similarity is defined as the weighted average of the cosine similarity of the feature vectors. Since the license plate may be blocked, the f of some sample points p Not available, w a is the overall visual feature of the vehicle a The corresponding weight, w d is the spatiotemporal feature f d The corresponding weight, w p is the license plate visual feature f p The corresponding weight.

[0069] The process of obtaining the multimodal features of each vehicle is as follows: Figure 2 As shown in the figure, each time a picture corresponding to a visual feature to be classified is input, the first step is to perform a coarse-grained search, searching its K nearest neighbor samples in the entire image library according to the visual feature, where K is a parameter. The second step is to perform precise classification, check the cluster where the K nearest neighbor samples are located, and calculate the multimodal similarity using the cluster center feature and the target sample feature. If the maximum similarity is greater than the threshold, the sample is classified into the corresponding class, otherwise the sample is treated as a new class alone, and finally the multimodal features of each vehicle can be obtained.

[0070] In large-scale practical application scenarios, the amount of recorded data generated by multiple vehicles under multiple cameras is very large, and there is a problem of unknown number of categories. Through the method of the above embodiment, the multimodal features of each vehicle are gradually improved according to the input of the sample. On the basis of obtaining K nearest neighbors, only K multimodal similarities need to be calculated to complete one-step classification. Moreover, since the role of K nearest neighbors is coarse-grained search, it does not need to be too precise. We use PCA dimensionality reduction for visual features and then search in parallel, which ensures the efficiency of obtaining multimodal features of each vehicle and reduces the space overhead of the calculation process.

[0071] The method of the above embodiment determines the multimodal similarity between the visual features of multiple vehicles and the spatiotemporal features of the vehicles, that is, on the basis of the visual features of the vehicles, it also fully considers the spatiotemporal rationality constraints of the trajectory, so that the calculation result of the multimodal similarity is more accurate, and the accuracy of the multimodal similarity calculation result is improved; and then according to the multimodal similarity between the visual features and the first threshold, the multimodal features of each vehicle can be accurately obtained, so that the final restored path is more accurate and reasonable.

[0072] Optionally, the path probability model is determined based on a trained path prior probability model and a trained path likelihood probability model; the path prior probability model is trained based on first training data; the first training data includes: information on each path between a first trajectory sampling point and a second trajectory sampling point passed by each vehicle in the historical trajectory; the path likelihood probability model is trained based on second training data; the second training data includes: vehicle speed information on each path between a first trajectory sampling point and a second trajectory sampling point passed by each vehicle in each time period in the historical trajectory.

[0073] Specifically, the path probability model is determined based on the trained path prior probability model and the trained path likelihood probability model; wherein the training data corresponding to the path prior probability model includes information about each path between the first trajectory sampling point and the second trajectory sampling point passed by each vehicle in the historical trajectory, and optionally, the information about each path includes probability information about each path between the first trajectory sampling point and the second trajectory sampling point passed by each vehicle in the historical trajectory; that is, the path prior probability model accurately describes the prior probability of the driver's path-finding preference by statistically analyzing the frequency of the driver's transfer between road sections in the historical trajectory data. rate; the training data corresponding to the path likelihood probability model includes the speed information of each vehicle passing through each path between the first trajectory sampling point and the second trajectory sampling point in each time period in the historical trajectory, that is, the path likelihood probability model calculates the average speed of each road in the road network in different time slices through the historical trajectory data to support likelihood probability modeling, and accurately describes the likelihood probability of the degree of consistency between the vehicle speed and the real-time road conditions; that is, the path probability model not only statistically describes the prior probability of the driver's path-finding preference, but also statistically describes the likelihood probability of the degree of consistency between the vehicle speed and the real-time road conditions, making the trajectory recovery result based on the path probability model more accurate and reasonable.

[0074] Optionally, the path prior probability model is determined using the following formula (1):

[0075]

[0076] Among them, r s represents the first trajectory sampling point; r erepresents the second trajectory sampling point; p represents any path between the first trajectory sampling point and the second trajectory sampling point; the path p is composed of the road segment s i Composition, p = {s1, ..., s n}; Pr(p|r s , r e ) represents the prior probability of the vehicle passing through path p; Pr(s i+1 |s i , r e ) indicates that the sampling point of the second trajectory is r e , the current section is s i When , the next section is s i+1 The prior probability of s , r e ) indicates that the sampling point of the second trajectory is r e , the probability that the first road segment is s1.

[0077] Specifically, since traffic cameras are not deployed at all intersections, and due to the camera field of view or the success rate of detecting vehicle objects from the video, the vehicle trajectory is not recorded when passing through every intersection. Vehicle trajectory recovery is to infer the most likely path of the vehicle moving between each two trajectory points corresponding to the clustering results that are adjacent in time order.

[0078] We have a given starting point r s , departure time t s , arrive at point r e , arrival time t e Under the condition that the path is composed of a series of road segments s i The composition p={s1,...,s n}, and conduct path probability modeling. Let Δt = t e -t s ,have:

[0079] Pr(p|r s , t s , r e , t e )=Pr(p|r s , Δt, r e , t e )∝Pr(p,Δt|r s , r e , t e )

[0080] =Pr(p|r s , r e , t e )Pr(Δt|p,t e )≈Pr(p|rs , r e )Pr(Δt|p,t e )

[0081] The approximation here is because when r is given s , r e When p and t e is weakly correlated.

[0082] The first item in the above probability decomposition is the prior item, that is, when the driver wants to go from r s Go to e When t is reached, how likely is it to choose path p in general? The second term is the likelihood term, that is, e The possibility that the time taken to travel path p is Δt under the real-time road conditions corresponding to the time.

[0083] For the prior term, we first obtain the prior probability distribution of the driver's road transfer at the intersection when the destination is given based on the prior probability distribution of the road transfer between forks and the statistics of the historical trajectory transfer between paths. Then, based on the Markov assumption, the prior term is calculated as:

[0084]

[0085] Where Pr(s i+1 |s i , r e ) is the prior probability distribution of the driver's road transfer at the intersection obtained based on the historical trajectory statistics, that is, at a given destination r e and the current road segment i When the next segment is selected, s i+1 The probability of Pr(s1|r s , r e ) is at the starting intersection, given the destination r e The probability of choosing s1 as the first segment. Since there is no predecessor segment at the start, Pr(s1|r s , r e ) is calculated for a given destination r e When , the sum of the transition probabilities of entering the intersection from any section and leaving the intersection from section s1.

[0086] The method of the above embodiment can accurately determine the probability of each vehicle passing through each path through a path prior probability model, that is, based on the probability of the vehicle selecting each path between each trajectory point in the historical trajectory, thereby providing an accurate and reasonable basis and determination method for vehicle path recovery, making the result of vehicle trajectory recovery more accurate.

[0087] Optionally, the path likelihood probability model is determined using the following formula (2):

[0088]

[0089] Where p represents any path between the first trajectory sampling point and the second trajectory sampling point; r s represents the first trajectory sampling point; r e represents the sampling point interval of the second trajectory; Indicates that at time t e The total expected travel time of the vehicle passing through each path between the first trajectory sampling point and the second trajectory sampling point in the historical trajectory; σ represents the standard deviation of the travel time of the vehicle passing through each path between the first trajectory sampling point and the second trajectory sampling point in the historical trajectory; Pr(Δt|p, t e ) indicates that at time t e Under this condition, the likelihood probability that the vehicle passes through path p in a travel time of Δt.

[0090] Specifically, for the path likelihood probability model, we first calculate the average vehicle speed of each road in the road network at each time slice (1h) based on the historical trajectory data to obtain the real-time expected travel time of the road. After obtaining the total expected travel time of each road section at time te for path p, we can get After that, assuming that the road travel time conforms to the Gaussian distribution and estimating the standard deviation σ from the historical trajectory, the likelihood term is calculated as:

[0091]

[0092] According to the above path probability model, the most likely path can be determined for a given starting and ending point and the starting and ending time: s Start from the existing path, extend one section outward each time, and keep the k paths with the highest current probability until reaching r e .

[0093] The method of the above embodiment can accurately determine the probability of each vehicle passing through each path through a path posterior probability model, that is, based on the travel time of each path in each time period in the historical trajectory data, so as to provide an accurate and reasonable basis and determination method for vehicle path recovery, so that the result of vehicle trajectory recovery is more accurate.

[0094] Optionally, after obtaining the trajectory recovery result of each vehicle according to the path probability model and the multimodal characteristics of each vehicle, the method further includes:

[0095] According to the trajectory recovery results of each vehicle, the noise sampling points and missing sampling points in the trajectory sampling points corresponding to the multimodal features of each vehicle are obtained;

[0096] Update the spatiotemporal features according to the noise sampling points and the missing sampling points;

[0097] According to the path probability model and the data of the updated trajectory sampling points of each vehicle, the target trajectory recovery result of each vehicle is obtained; the data of the updated trajectory sampling points of each vehicle is determined based on the updated spatiotemporal features.

[0098] Specifically, for the vehicle trajectory recovery task, the acquisition of multimodal features (i.e., vehicle clustering) is an upstream task of trajectory recovery. Trajectory recovery infers the path between every two adjacent trajectory points based on the clustering results. It can be seen that when the clustering result is incorrect, the correctness of the recovered trajectory is naturally greatly affected. Conversely, when the recovered trajectory is unreasonable, it also indicates that the clustering result may be wrong. Based on the above problems, the embodiment of the present application discovers the possible noise and omissions in the clustering results based on the spatiotemporal rationality constraints of the restored trajectory, and iterates the trajectory recovery scheme by updating the spatiotemporal features of the clustering input. That is, according to the trajectory recovery results of each vehicle, the noise sampling points and missing sampling points in the restored trajectory of each vehicle can be obtained, and the spatiotemporal features can be updated according to the noise sampling points and missing sampling points to complete the update and iteration of the spatiotemporal features. Then, according to the iterated spatiotemporal features, the multimodal features can be updated and iterated, and the update and iteration of the trajectory sampling points can be realized; finally, by updating the iterative trajectory sampling points and path probability model, the updated trajectory recovery results can be obtained, that is, the noise sampling points and missing sampling points are obtained based on the vehicle trajectory recovery results, and the spatiotemporal features and multimodal features are iterated, so that the iterative trajectory recovery results are finally made more accurate and reasonable.

[0099] For example, the vehicle trajectory recovery process of the embodiment of the present application is as follows: Figure 3As shown, first, vehicles are clustered based on spatiotemporal features and vehicle visual features, and then the vehicle trajectories are restored based on the clustering results; further, the embodiment of the present application also evaluates the recovery results of the vehicle trajectories, and the feedback module updates and iterates the spatiotemporal features based on the evaluation results, thereby realizing the iteration of the restored trajectories, and finally making the output vehicle trajectory recovery results after iteration more accurate and reasonable. It should be emphasized that the vehicle trajectory recovery process in the embodiment of the present application mainly includes a vehicle identification clustering process, a trajectory path recovery process, and a feedback process for extracting the spatiotemporal constraints of the restored trajectory and inputting them into the clustering. Through the feedback (iteration) mechanism, an iterative framework is formed. Based on more correct clustering results, more realistic trajectories are restored. Based on more realistic trajectories, more accurate spatiotemporal constraint information is extracted and clustering is guided. The collaborative iterative improvement of the two tasks of vehicle identification clustering and trajectory path recovery is achieved. That is, the multimodal features of the vehicle are first obtained, and clustering based on multimodal similarity is used to distinguish different vehicles and find different trajectory points of the same vehicle; then, based on the vehicle identification clustering results, that is, different trajectory points of the same vehicle, the maximum probability path of the vehicle between trajectory points is restored based on path probability modeling; finally, according to the spatiotemporal rationality constraints of the restored trajectory, the possible noise and missing in the clustering results are found. By updating the spatiotemporal features of the clustering input, the entire system forms an iterative framework, so that the spatiotemporal features and vehicle trajectory recovery form a feedback and iterative mechanism, so that the vehicle trajectory restored by the iterative mechanism is more accurate. It should be noted that vehicle clustering is an upstream task of trajectory recovery. Trajectory recovery infers the path between each two adjacent trajectory points based on the clustering results. It can be seen that when the clustering results are incorrect, the correctness of the restored trajectory is naturally greatly affected. Conversely, when the recovered trajectory is unreasonable, it also indicates that the clustering result may be wrong. Therefore, we designed a feedback mechanism and feedback module to discover the possible noise and omissions in the clustering results based on the spatiotemporal rationality constraints of the recovered trajectory, and to form an iterative framework for the entire system by updating the spatiotemporal constraint features of the clustering input. Optionally, the noise sampling points and missing sampling points in the trajectory sampling points are obtained based on the vehicle trajectory recovery results, and the iteration of the spatiotemporal features and the vehicle recovery trajectory can be implemented according to the following process, that is, according to the trajectory recovery results of each vehicle, the noise sampling points in the trajectory sampling points corresponding to the multimodal features of each vehicle are obtained:

[0100] If all trajectory points in a certain class of a given clustering result R = {r1, ..., r n}, let all possible trajectories passing through R be For any of the trajectories We define a function f to evaluate its rationality f(P), and define the rationality of R f(R) as The rationality of the most rational trajectory in :

[0101]

[0102] When there are noisy sampling points in R, since the recovered trajectory must pass through trajectory points that do not belong to the vehicle, it is likely to result in a low rationality. Therefore, we look for the optimal subset R of R * So that f(R * ) is the largest, and R is not in the optimal subset R * The points are detected as noise:

[0103]

[0104] The trajectory rationality evaluation function f(P) is defined as follows: for R′={r′1, ..., r′ m}, the trajectory P is divided into m-1 sub-trajectories by these m trajectory points: p1, ..., p m-1 , for each sub-trajectory p i , using the trajectory recovery probability model, calculate its probability Pr(p i |r si , t si , r ei , t ei ), for simplicity, recorded as Pr(p i ), then f(P) is defined as the geometric mean of the probabilities of these sub-trajectories with a certain adjustment:

[0105]

[0106] in The term is a compensation for the subset size m, so that as many points as possible are included on the basis of ensuring that no obvious noise sampling points appear in the subset; otherwise, a pair of points in R with the highest probability of moving subtrajectories between each other will become the optimal subset.

[0107] Finally, for the detected noise sampling points, update their spatiotemporal features in the next iteration to make them away from the average features of the non-noise sampling points in this class. λ is the update rate parameter:

[0108]

[0109] Optionally, the missing sampling points can be obtained by the following steps:

[0110] Iteration is achieved through missing sample point recall, where the goal of missing sample point recall is to recall the missing samples in the class corresponding to a certain car, that is, the samples that are misclassified into the class of other cars. We call the two main missing situations point missing and block missing, and deal with them separately.

[0111] Point missing means that there are sporadic missing points in the trajectory points. The optimal path inferred by trajectory recovery is likely to still correctly pass through the intersection corresponding to the missing sampling point. Therefore, whenever the recovered trajectory passes through an intersection with a traffic camera, but there is no corresponding trajectory point in the clustering result, it is determined whether a missing point has occurred.

[0112] Specifically, the noise sampling points detected from all classes are used as candidate libraries for missing recall; when a point is found missing in a class, the previous point of the possible missing sampling point is recorded as r i-1 , the next point is r i+1 , select from the candidate library the visual feature similarity greater than a certain threshold and within r i-1 to r i+1 The point passing through the missing intersection within the time range is r i Considering that there is no corresponding record when a vehicle passes by the camera, it may be caused by the camera's field of view or the success rate of detecting vehicle objects from the video, not necessarily due to the lack of clustering. The probability that the camera successfully records a car passing by it is denoted as p c .

[0113] Define a hypothesis H0 that the cluster is not missing, but the camera failed to record successfully. The confidence level is:

[0114] g(H0)=Pr(r i+1 |r i-1 )(1-p c )

[0115] Define another hypothesis H1: the camera successfully recorded, but the clustering was missing. The confidence is:

[0116] g(H1)=Pr(r i |r i-1 )Pr(r i+1 |r i ) c

[0117] When g(H1)>g(H0), we accept hypothesis H1, and in the next iteration, the corresponding missing sampling point r i The spatiotemporal constraint features of are updated to the average value of the class:

[0118]

[0119] Block missing means that a vehicle trajectory is dispersed into several classes in blocks. In each class, the block-like trajectory points themselves can constitute a subset with a large degree of rationality, so they will not be detected as noise and cannot be recalled by a method similar to recalling point missing. We consider recalling block missing by absorbing some small classes into large classes. Specifically, first, for each large class with a large number of trajectory points, search for several small classes with a small number of candidate trajectory points according to the similarity threshold of visual features. Then try to merge these small classes into the large class, and perform noise detection on the fused new class. If the newly added point is not detected as noise, it will be recalled to the large class, and its spatiotemporal constraint features will be updated to the average value of the large class. Exemplarily, the detection of noise sampling points and missing sampling points is as follows. Figure 4 shown.

[0120] The method of the above embodiment obtains the noise sampling points and missing sampling points in the trajectory sampling points corresponding to the multimodal features of each vehicle according to the trajectory recovery results of each vehicle, and updates the spatiotemporal features according to the noise sampling points and missing sampling points, thereby improving the accuracy and rationality of the multimodal features, i.e., the clustering of each vehicle. After multiple iterations of trajectory recovery, update iterations of spatiotemporal features, and iterations of vehicle clustering, the detected noise sampling points and missing sampling points gradually decrease, so that the trajectory recovery results of each vehicle are more accurate and reasonable.

[0121] Exemplarily, the vehicle trajectory recovery method and system of the embodiment of the present application are as follows: Figure 5 shown.

[0122] First, we preprocess the video data and historical trajectory data. For video data, we use target detection technology to obtain the images recorded when each vehicle passes each camera, and use convolutional neural network technology to extract visual features, including the feature vector of the vehicle's overall appearance and the feature vector of the license plate area. For the collected historical GPS trajectory data, we use the road network matching algorithm to match the downloaded road network and GPS trajectory data to obtain the corresponding road network trajectory data (road network trajectory for each time slice).

[0123] Then, vehicle recognition clustering is performed through the multimodal similarity vehicle clustering module. The extracted vehicle appearance features and license plate features, as well as a spatiotemporal feature that is dynamically updated with iterations, are obtained to obtain vehicle similarity, license plate similarity, spatiotemporal constraint similarity, and the similarity threshold after the three are combined. The images recorded by each vehicle when passing through different cameras are clustered to achieve clustering of each vehicle based on the vehicle's visual features and spatiotemporal constraint features.

[0124] Finally, the clustering results of each vehicle are used as vehicle trajectory sampling points (trajectory points corresponding to the clustering results), and the maximum a posteriori probability search is implemented according to the path probability model and the data of the trajectory sampling points of each vehicle to obtain the trajectory recovery results of each vehicle (vehicle road network movement trajectory). The path probability model describes the likelihood probability of the degree of conformity between the vehicle speed and the real-time road conditions and the prior probability of the driver's path-finding preference, so that the vehicle trajectory recovery results obtained according to the path probability model are more accurate. Optionally, the average vehicle speed of each road in the road network in different time slices is calculated according to the historical trajectory to support likelihood probability modeling, that is, the road network trajectory of each time slice is matrix decomposed to realize road speed estimation, and the average road speed of each time slice is obtained; on the other hand, the frequency of driver transfer between road sections (intersection entry and exit transfer probability distribution) is counted to support prior probability modeling, that is, the road network trajectory of each time slice is frequency counted to realize transfer probability estimation, and the intersection entry and exit transfer probability distribution is obtained. In the process of path recovery in the embodiment of the present application, a highly interpretable Bayesian path probability model is established by comprehensively considering the rationality of time and space and the driver's habits, so that the result of path recovery is more accurate and reasonable.

[0125] Furthermore, an iterative framework is formed through the feedback module. The feedback module detects potential noise sampling points (noise detection) and missing sampling points (missing recall) in the clustering results by evaluating the spatiotemporal rationality of the restored trajectory, and combines the spatiotemporal constraint characteristics of vehicle clustering to improve the clustering effect in the next round of iteration. After several rounds of collaborative iterations of vehicle clustering and road network trajectory recovery, the detected noise sampling points and missing sampling points gradually decrease. At this time, the restored continuous moving trajectory of each vehicle on the road network is the final output of the system. The feedback module detects noise sampling points and missing sampling points based on the quantitative evaluation of the rationality of the trajectory, and updates the spatiotemporal characteristics to make the path trajectory recovery result more accurate.

[0126] The embodiment of the present application fully considers the temporal and spatial rationality constraints of the trajectory, which not only improves the accuracy of path recovery, but also enables clustering to directly utilize this constraint information, thereby improving the clustering effect compared to clustering based on pure visual features; the system forms an iterative framework through a feedback mechanism, recovers more realistic trajectories based on more correct clustering results, extracts more accurate temporal and spatial constraint information based on more realistic trajectories and guides clustering, thereby achieving coordinated iterative improvement of the two tasks of vehicle recognition clustering and trajectory path recovery.

[0127] For example, Figure 6 As shown, another embodiment of the present application is as follows:

[0128] In order to use the video data recorded by all traffic cameras in a certain district of a city in one day, obtain all representative images of each vehicle captured by the camera, and its continuous movement trajectory on the road network represented by the end-to-end road segments, the system can be implemented as follows:

[0129] (1) Data preprocessing;

[0130] A. Road network construction and processing;

[0131] You can directly use the road network built by the traffic management department or geographic map service company, or you can download the road network data of the city or district from the public map data source and match the traffic cameras to the intersection nodes in the road network.

[0132] B. Video data processing;

[0133] Through frame extraction, vehicle target detection and visual feature extraction, the event of a car passing through the camera is converted into a representative image and corresponding features, the small frame where the vehicle is located is detected, and feature vectors of a certain dimension are extracted for the vehicle and license plate respectively. 256 dimensions are recommended. When the license plate is blocked, the missing license plate feature can be allowed.

[0134] C. Trajectory data processing;

[0135] In order to obtain the average road speed and the driver's route-finding habits, it is necessary to estimate and learn from a certain amount of historical trajectories in the city and area. For example, data from taxi GPS driving trajectory records, or anonymous user data recorded by navigation service-related companies. The trajectory data obtained is generally a sequence consisting of GPS and timestamps. By applying a road network matching algorithm, it is matched to the road network and converted into a sequence of transfers between continuous road sections.

[0136] (2) Path probability modeling;

[0137] D. Prior probability modeling;

[0138] Based on the historical trajectories after road network matching, the path transfer probability distribution of drivers at intersections can be modeled. Specifically, all trajectories to the same destination are filtered out from the historical trajectories (the trajectory passing through a destination can be truncated here, and the trajectory before the truncation is regarded as the trajectory to the destination), and then all drivers going to the destination are counted. When entering a certain intersection from a certain road, the frequency of which road they choose to exit is counted to model the prior probability of the trajectory path.

[0139] E. Likelihood probability modeling;

[0140] Based on the historical trajectory after road network matching, the speed of vehicles on the road can be directly calculated. In order to take into account the time-varying nature of road conditions, it is necessary to estimate the average speed of the road in time slices. Considering that when the amount of historical data is small, some roads may lack speed estimates at certain times. It is possible to consider increasing the length of the time slice appropriately and using the classic matrix decomposition method to complete the speed estimation matrix. It is also possible to further improve the road speed estimation results by directly collecting public road speed data or deploying speed measuring equipment to measure, and then model the likelihood probability of the trajectory path. Among them, about 100,000 valid historical trajectories can support the speed estimation of roads in a certain area of ​​the city at a 1h time slice granularity.

[0141] (3) Vehicle trajectory recovery;

[0142] F. Vehicle identification clustering;

[0143] The recorded images of all vehicles passing through all traffic cameras obtained in preprocessing are used as the input samples of the clustering algorithm, and all images of each vehicle are clustered. The feature of each sample is the vehicle visual feature f obtained in preprocessing. a , license plate visual features p , and initialized to f a The spatiotemporal characteristics of d The weights between features are parameters that need to be adjusted. The recommended value is w a :w p :w d =1∶8∶1, and the weight can also be adjusted appropriately according to the practical effect. The clustering threshold of feature similarity can generally be set to 0.8, which can be adjusted appropriately according to the practical effect.

[0144] G. Road network trajectory recovery;

[0145] Each class in the clustering result corresponds to a vehicle, and each sample image corresponds to a trajectory point. In chronological order, the optimal path search algorithm based on trajectory probability modeling is implemented between every two adjacent trajectory points to restore the continuous road network trajectory of the vehicle moving between the trajectory points. Among them, the parameter k in the search algorithm is the number of optimal paths retained after each path growth, and it is generally set to 5 to ensure that the optimal path is searched.

[0146] H. Feedback iteration to improve efficiency;

[0147] Noise detection and missing recall are performed, and the spatiotemporal features are updated according to the spatiotemporal rationality of the restored trajectory. The update rate parameter λ of the spatiotemporal features is generally set to 0.2; and the probability p of the camera successfully capturing a vehicle passing by it cIt is usually set to around 0.7. You can adjust it based on the camera's imaging quality and the target detection effect during preprocessing. If conditions permit, you can even manually inspect some samples and count the percentage of successful camera shots. You can also set different p values ​​for cameras with different imaging quality and shooting conditions. c value.

[0148] Generally speaking, 5 iterations can achieve the best effect of the system. Optionally, you can obtain some clustering truth data and trajectory data by driving a vehicle or collecting some real trips of vehicles on the day of video data collection, so as to evaluate the effect of the system. Generally, commonly used evaluation indicators such as precision, recall rate, and f-score, which comprehensively evaluates precision and recall rate, can be used as evaluation criteria for clustering. When clustering achieves the best effect, the iteration is terminated. It can be assumed that trajectory recovery as a downstream task will also achieve the best effect. At this time, the obtained clustering results and the recovered trajectory are the target output.

[0149] Exemplarily, another embodiment of the present application is as follows:

[0150] In order to use a picture of a suspected vehicle, query all the pictures of the suspected vehicle recorded by the camera from all the traffic camera video data retrieved from the approximate area of ​​the vehicle's activity, and restore its movement path, the following steps can be implemented:

[0151] (1) Vehicle search;

[0152] The target vehicle can be retrieved by processing the video data to obtain a picture library and its corresponding visual features, and similarly extracting visual features from the query image of the suspected vehicle. Then, based on the multimodal feature similarity threshold, all images that may be of the suspected vehicle are retrieved.

[0153] (2) Trajectory recovery;

[0154] The optimal path search based on path probability modeling is implemented for the trajectory points corresponding to the above pictures to obtain the most likely movement path of the vehicle on the road network.

[0155] (3) Iteration;

[0156] Perform noise sampling point detection on the recovered trajectory and remove the noise sampling points from the retrieval results. Perform missing sampling point detection on the recovered trajectory. Optionally, the similarity threshold can be lowered to retrieve more candidate images from the image library and try to add them to the existing retrieval results. Perform noise detection and absorb the images that are not detected as noise.

[0157] After modifying the retrieval results, re-perform trajectory recovery, and then perform noise detection and missing recall again, iterating several times until the results are stable. Optionally, since you are only interested in a specific target vehicle, you can also perform manual noise detection directly to remove pictures that are not the target vehicle in the retrieval results.

[0158] Exemplarily, another embodiment of the present application is as follows:

[0159] Given several time-space trajectory points of the target vehicle, but not its appearance, in order to recover the trajectory of the target vehicle and find the vehicle that matches the trajectory of the vehicle from the traffic camera data, so as to know the appearance of the vehicle, the following steps can be implemented:

[0160] (1) Track point matching;

[0161] If the location of the trajectory point is provided in the form of GPS coordinates, it can be simply projected onto the nearest road in the road network, or a road network matching algorithm can be applied similar to the preprocessing of historical trajectory data. Optionally, assuming that the traffic camera is deployed at an intersection, that is, a node in the road network, if the trajectory point is known to be in the middle of the road, the road segment can be split into two segments at this point when constructing the road network.

[0162] (2) Target vehicle trajectory recovery;

[0163] The optimal path search based on path probability modeling is implemented for the above trajectory points to obtain the most likely movement path of the vehicle on the road network.

[0164] (3) Recover the trajectories of all vehicles in the traffic camera data;

[0165] (4) Searching for vehicles that may be the target vehicle from the camera data;

[0166] The vehicle trajectory recovered from the camera data is matched with the target vehicle trajectory. The vehicle image corresponding to the trajectory with a high matching degree can be regarded as the appearance of the suspected target vehicle.

[0167] For example, we collected 24 hours of video data from 441 traffic cameras in a certain area, processed the recorded data generated when 4 million vehicles passed by the cameras, and obtained a road network with 2966 edges and 1263 nodes, and collected 147661 historical trajectories and the true trajectories of 197 vehicles on the same day. The system achieved 86.6% precision and 82.6% recall rate in clustering effect, and improved the final trajectory recovery effect by about 10%.

[0168] Figure 7The trajectory recovery results corresponding to the trajectory recovery method are shown. The figure shows the recorded images of a vehicle passing through 10 traffic cameras on the same day, as well as the clustering and trajectory recovery results. The black solid line is the recovered trajectory, and the black arrow trajectory is the true trajectory. It can be seen that the system completely and accurately clustered these 10 records and almost completely correctly restored the vehicle's movement path.

[0169] The vehicle trajectory recovery device provided by the present invention is described below. The vehicle trajectory recovery device described below and the vehicle trajectory recovery method described above can be referred to each other.

[0170] Figure 8 : is a schematic diagram of the structure of the vehicle trajectory recovery device provided by the present invention. The vehicle trajectory recovery device provided in this embodiment includes:

[0171] The acquisition module 710 is used to acquire visual features of multiple vehicles obtained based on multiple traffic cameras; acquire multimodal features of each vehicle according to the visual features of the multiple vehicles and the spatiotemporal features of the trajectory constraints on the vehicles; and acquire data of multiple trajectory sampling points of each vehicle according to the multimodal features of each vehicle;

[0172] The recovery module 720 is used to obtain the trajectory recovery result of each vehicle according to the path probability model and the data of the trajectory sampling points of each vehicle; the path probability model is used to determine the path with the maximum probability of each vehicle between adjacent trajectory sampling points.

[0173] Optionally, the acquisition module 710 is specifically used to: determine the multimodal similarity between the visual features of each vehicle according to the visual features of multiple vehicles and the spatiotemporal features of the vehicles;

[0174] The multimodal features of each vehicle are obtained according to the multimodal similarity between the visual features of each vehicle and a first threshold.

[0175] Optionally, the path probability model is determined based on a trained path prior probability model and a trained path likelihood probability model; the path prior probability model is trained based on first training data; the first training data includes: information on each path between a first trajectory sampling point and a second trajectory sampling point passed by each vehicle in the historical trajectory; the path likelihood probability model is trained based on second training data; the second training data includes: vehicle speed information on each path between a first trajectory sampling point and a second trajectory sampling point passed by each vehicle in each time period in the historical trajectory.

[0176] Optionally, the acquisition module 710 is specifically used to determine the path prior probability model using the following formula (1):

[0177]

[0178] Among them, r s represents the first trajectory sampling point; r e represents the second trajectory sampling point; p represents any path between the first trajectory sampling point and the second trajectory sampling point; the path p is composed of the road segment s i Composition, p = {s1, ..., s n}; Pr(p|r s , r e ) represents the prior probability of the vehicle passing through path p; Pr(s i+1 |s i , r e ) indicates that the sampling point of the second trajectory is r e , the current section is s i When , the next section is s i+1 The prior probability of s , r e ) indicates that the sampling point of the second trajectory is r e , the probability that the first road segment is s1.

[0179] Optionally, the acquisition module 710 is specifically used to determine the path likelihood probability model using the following formula (2):

[0180]

[0181] Where p represents any path between the first trajectory sampling point and the second trajectory sampling point; r s represents the first trajectory sampling point; r e represents the sampling point interval of the second trajectory; Indicates that at time t e The total expected travel time of the vehicle passing through each path between the first trajectory sampling point and the second trajectory sampling point in the historical trajectory; σ represents the standard deviation of the travel time of the vehicle passing through each path between the first trajectory sampling point and the second trajectory sampling point in the historical trajectory; Pr(Δt|p, t e ) indicates that at time t e Under this condition, the likelihood probability that the vehicle passes through path p in a travel time of Δt.

[0182] Optionally, the recovery module 720 is further used to: obtain noise sampling points and missing sampling points in the trajectory sampling points corresponding to the multimodal features of each vehicle according to the trajectory recovery results of each vehicle;

[0183] Update the spatiotemporal features according to the noise sampling points and the missing sampling points;

[0184] According to the path probability model and the data of the updated trajectory sampling points of each vehicle, the target trajectory recovery result of each vehicle is obtained; the data of the updated trajectory sampling points of each vehicle is determined based on the updated spatiotemporal features.

[0185] The device of the embodiment of the present invention is used to execute the method in any of the aforementioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0186] Fig. 9 An example of a physical structure diagram of an electronic device is provided, and the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a vehicle trajectory recovery method, and the method includes: obtaining visual features of multiple vehicles obtained based on multiple traffic cameras; obtaining multimodal features of each vehicle according to the visual features of the multiple vehicles and the spatiotemporal features of the trajectory constraints of the vehicles; obtaining data of multiple trajectory sampling points of each vehicle according to the multimodal features of each vehicle; obtaining trajectory recovery results of each vehicle according to the path probability model and the data of the trajectory sampling points of each vehicle; the path probability model is used to determine the path with the maximum probability of each vehicle between adjacent trajectory sampling points.

[0187] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0188] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the vehicle trajectory recovery method provided by the above-mentioned methods, and the method includes: obtaining visual features of multiple vehicles obtained based on multiple traffic cameras; obtaining multimodal features of each vehicle based on the visual features of the multiple vehicles and the spatiotemporal features of the trajectory constraints on the vehicles; obtaining data of multiple trajectory sampling points of each vehicle based on the multimodal features of each vehicle; obtaining the trajectory recovery result of each vehicle based on the path probability model and the data of the trajectory sampling points of each vehicle; the path probability model is used to determine the path with the maximum probability of each vehicle between adjacent trajectory sampling points.

[0189] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the vehicle trajectory recovery method provided above, the method comprising: obtaining visual features of multiple vehicles obtained based on multiple traffic cameras; obtaining multimodal features of each vehicle based on the visual features of the multiple vehicles and the spatiotemporal features of the trajectory constraints on the vehicles; obtaining data of multiple trajectory sampling points of each vehicle based on the multimodal features of each vehicle; obtaining trajectory recovery results of each vehicle based on a path probability model and the data of the trajectory sampling points of each vehicle; the path probability model is used to determine the probability of the vehicle passing through at least one path between a first trajectory sampling point and a second trajectory sampling point among the trajectory sampling points.

[0190] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0191] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle trajectory recovery method, characterized in that: include: Obtain visual features of multiple vehicles based on multiple traffic cameras; Acquire multimodal features of each of the vehicles according to visual features of the multiple vehicles and spatiotemporal features of trajectory constraints on the vehicles; Acquiring data of a plurality of trajectory sampling points of each of the vehicles according to the multimodal characteristics of each of the vehicles; According to the path probability model and the data of the trajectory sampling points of each of the vehicles, the trajectory recovery results of each of the vehicles are obtained; the path probability model is used to determine the path with the maximum probability of each vehicle between adjacent trajectory sampling points; The visual features of the vehicle include at least one of the following: an overall visual feature of the vehicle and a visual feature of a license plate of the vehicle; Acquiring multimodal features of each of the vehicles according to the visual features of the multiple vehicles and the spatiotemporal features of the vehicles, including: Determining a multimodal similarity between visual features of each of the vehicles based on the visual features of the multiple vehicles and the spatiotemporal features of the vehicles; Acquire the multimodal features of each of the vehicles according to the multimodal similarity between the visual features of each of the vehicles and a first threshold; The path probability model is determined based on a trained path prior probability model and a trained path likelihood probability model; the path prior probability model is trained based on first training data; the first training data includes: information on each path between a first trajectory sampling point and a second trajectory sampling point passed by each vehicle in a historical trajectory; the path likelihood probability model is trained based on second training data; the second training data includes: vehicle speed information on each path between the first trajectory sampling point and the second trajectory sampling point passed by each vehicle in each time period in the historical trajectory.

2. The vehicle trajectory recovery method according to claim 1, characterized in that: The path prior probability model is determined using the following formula (1): Among them, r s represents the first trajectory sampling point; r e represents the second trajectory sampling point; p represents any path between the first trajectory sampling point and the second trajectory sampling point; the path p is composed of the road segment s i Composition, p={s1,…,s n }; Pr(p|r s ,r e ) represents the prior probability of the vehicle passing through path p; Pr(s i+1 |s i ,r e ) indicates that the sampling point of the second trajectory is r e , the current section is s i When , the next section is s i+1 The prior probability of s ,r e ) indicates that the sampling point of the second trajectory is r e , the probability that the first road segment is s1.

3. The vehicle trajectory recovery method according to claim 2, characterized in that: The path likelihood probability model is determined using the following formula (2): Where p represents any path between the first trajectory sampling point and the second trajectory sampling point; r s represents the first trajectory sampling point; r e represents the sampling point of the second trajectory; Indicates that at time t e The total expected travel time of the vehicle passing through each path between the first trajectory sampling point and the second trajectory sampling point in the historical trajectory; σ represents the standard deviation of the travel time of the vehicle passing through each path between the first trajectory sampling point and the second trajectory sampling point in the historical trajectory; Pr(Δt|p,t e ) indicates that at time t e Under this condition, the likelihood probability that the vehicle passes through the path p in a travel time of Δt.

4. The vehicle trajectory recovery method according to any one of claims 1 to 3, characterized in that: After obtaining the trajectory recovery result of each vehicle according to the path probability model and the multimodal characteristics of each vehicle, the method further includes: According to the trajectory recovery results of each of the vehicles, noise sampling points and missing sampling points in the trajectory sampling points corresponding to the multimodal features of each of the vehicles are obtained; Updating the spatiotemporal features according to the noise sampling points and the missing sampling points; According to the path probability model and the data of the updated trajectory sampling points of each vehicle, the target trajectory recovery result of each vehicle is obtained; the data of the updated trajectory sampling points of each vehicle is determined based on the updated spatiotemporal features.

5. A vehicle trajectory recovery device, characterized in that: include: An acquisition module, used for acquiring visual features of multiple vehicles obtained based on multiple traffic cameras; According to the visual features of the multiple vehicles and the spatiotemporal features of the trajectory constraints on the vehicles, the multimodal features of each of the vehicles are obtained; according to the multimodal features of each of the vehicles, data of multiple trajectory sampling points of each of the vehicles are obtained; the visual features of the vehicles include at least one of the following: the overall visual features of the vehicle and the visual features of the license plate of the vehicle; Acquiring multimodal features of each of the vehicles according to the visual features of the multiple vehicles and the spatiotemporal features of the vehicles, including: determining multimodal similarities between the visual features of each of the vehicles according to the visual features of the multiple vehicles and the spatiotemporal features of the vehicles; acquiring multimodal features of each of the vehicles according to the multimodal similarities between the visual features of each of the vehicles and a first threshold; A recovery module is used to obtain trajectory recovery results of each vehicle according to a path probability model and data of trajectory sampling points of each vehicle; the path probability model is used to determine the path with the maximum probability of each vehicle between adjacent trajectory sampling points; the path probability model is determined based on a trained path prior probability model and a trained path likelihood probability model; the path prior probability model is trained based on first training data; the first training data includes: information on each path between a first trajectory sampling point and a second trajectory sampling point passed by each vehicle in a historical trajectory; the path likelihood probability model is trained based on second training data; the second training data includes: vehicle speed information on each path between the first trajectory sampling point and the second trajectory sampling point passed by each vehicle in each time period in the historical trajectory.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the vehicle trajectory recovery method according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the vehicle trajectory recovery method according to any one of claims 1 to 4 is implemented.

8. A computer program product having executable instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor implements the vehicle trajectory recovery method as described in any one of claims 1 to 4.

Images