Vehicle track recovery method and device based on unified imaging data modeling

Abstract

The invention relates to a vehicle track recovery method and device based on unified imaging data modeling, and the method comprises the steps: obtaining candidate points of a to-be-recovered track and a first image, and the first image comprises a topological structure of a road network where the to-be-recovered track is located; mapping the candidate points to the first image according to the position information of the candidate points, obtaining gray values of pixels corresponding to the candidate points in the first image according to the time information of the candidate points, and obtaining a second image; and predicting according to the second image to obtain a first trajectory prediction image of the trajectory to be recovered. By adopting the method, the problems of low vehicle track recovery precision and poor applicability can be solved.

Description

Technical Field

[0001] The present application relates to the field of trajectory recovery, and in particular to a vehicle trajectory recovery method and device based on unified imaging data modeling. Background Art

[0002] Trajectory recovery usually refers to a method of reconstructing the motion path of an object or entity based on incomplete or inaccurate data. In related technologies, clustering methods are usually used to generate trajectory results, or time series data are predicted through pre-trained models such as machine learning models and application sequence models to infer the most likely trajectory. However, the spatial constraints of the road network are crucial to trajectory inference, and the road network data used by related methods cannot fully express the complex spatiotemporal relationships required for trajectory sequences, and fail to make full use of traffic network data, resulting in inaccurate trajectory inference results obtained by trajectory recovery methods in complex or irregular road environments. This limitation is more significant in the recovery of low-precision positioning or non-continuous trajectories. At the same time, the road network data expressions of related technologies have their own limitations, and it is often necessary to design corresponding trajectory sequence recovery methods for specific road network data expressions, which results in the trajectory recovery effect being easily affected by strategies and unstable and having weak generalization.

[0003] With regard to the problems of low vehicle trajectory recovery accuracy and poor applicability in related technologies, no effective solution has been proposed so far. Summary of the invention

[0004] Based on this, it is necessary to provide a vehicle trajectory recovery method and device based on unified imaging data modeling to address the above technical problems, so as to solve the problems of low accuracy and poor applicability of vehicle trajectory recovery.

[0005] In a first aspect, a vehicle trajectory recovery method based on unified imaging data modeling is provided in this embodiment, and the method includes:

[0006] Obtain candidate points of the trajectory to be restored;

[0007] Obtaining a first image according to information of a road network where the trajectory to be restored is located;

[0008] Obtaining features of the candidate points on the first image according to the multimodal data of the candidate points, and combining the features with the first image to obtain a second image;

[0009] A first trajectory prediction image of the trajectory to be restored is obtained according to the second image.

[0010] In some embodiments, obtaining a feature of the candidate point on the first image according to the multimodal data of the candidate point, and combining the feature with the first image to obtain the second image includes:

[0011] Mapping the candidate point to the first image according to the position information of the candidate point;

[0012] Obtaining a grayscale value of the candidate point in the first image according to the time information of the candidate point;

[0013] The range of pixels occupied by the candidate point in the first image is obtained according to the reliability of the candidate point.

[0014] In some embodiments, the first network includes a first encoder and a decoder, the second network includes a second encoder and the decoder, and the predicting of the first trajectory prediction image of the trajectory to be restored according to the second image includes:

[0015] Acquire a third image generated according to the candidate points in the training trajectory, a positive sample image and a negative sample image containing the training trajectory;

[0016] Using the third image as an input of the first network and using the positive sample image as a supervision signal of the first network to train the first network;

[0017] Using the positive sample image as the input of the second network and the supervision signal of the second network, training the second network;

[0018] training the first encoder and the second encoder according to data output by the first encoder based on the third image and data output by the second encoder based on the positive sample image and the negative sample image respectively;

[0019] The first trajectory prediction image is obtained by predicting the first trained network according to the second image.

[0020] In some embodiments, the first network is trained by taking the third image as input and the positive sample image as a supervisory signal; the second network is trained by taking the positive sample image as input and a supervisory signal; and the first encoder and the second encoder are trained according to data output by the first encoder based on the third image and data output by the second encoder based on the positive sample image and the negative sample image, respectively, including:

[0021] Taking the third image as input and the positive sample image as a supervision signal, constructing a first loss function of the first network;

[0022] Taking the positive sample image as input and supervision signal, constructing a second loss function of the second network;

[0023] constructing a third loss function according to data output by the first encoder based on the third image and data output by the second encoder based on the positive sample image and the negative sample image respectively;

[0024] The first encoder, the second encoder, and the decoder are trained in combination with the first loss function, the second loss function, and the third loss function.

[0025] In some embodiments, training the first encoder, the second encoder, and the decoder in combination with the first loss function, the second loss function, and the third loss function includes:

[0026] Constructing a comprehensive loss value according to the first loss function, the second loss function and the third loss function;

[0027] When the comprehensive loss value satisfies the convergence condition, the training of the first encoder, the second encoder and the decoder is terminated.

[0028] In some embodiments, obtaining the grayscale value of the pixel corresponding to the candidate point in the first image according to the time information of the candidate point includes:

[0029] Obtaining a plurality of time information according to the timestamps of the plurality of candidate points;

[0030] Normalizing the plurality of time information;

[0031] The normalized time information is encoded to obtain the pixel grayscale value of the candidate point.

[0032] In some embodiments, after predicting the first trajectory prediction image of the trajectory to be restored according to the second image, the method further includes:

[0033] A road network trajectory sequence of the vehicle is obtained according to the degree of overlap between the first trajectory prediction image and the topological structure of the road network.

[0034] In some embodiments, after predicting the first image of the trajectory to be restored according to the second image, the method further includes:

[0035] Acquire a reference image of the trajectory to be restored;

[0036] The first image is evaluated based on an image difference between the reference image and the first image.

[0037] In a third aspect, a computer device is provided in this embodiment, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the vehicle trajectory recovery method based on unified imaging data modeling described in the first aspect is implemented.

[0038] In a fourth aspect, in this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the vehicle trajectory recovery method based on unified imaging data modeling described in the first aspect is implemented.

[0039] In a fifth aspect, a computer program product is provided in this embodiment, including a computer program, which, when executed by a processor, implements the vehicle trajectory recovery method based on unified imaging data modeling described in the first aspect above.

[0040] The above-mentioned vehicle trajectory recovery method and device based on unified imaging data modeling obtains a second image by fusing a first image containing a road network background and a candidate point of the trajectory, so that the second image contains the location information, time information and topological structure information of the trajectory in the road network background. Therefore, when recovering the trajectory, the second image can be fully used to extract various types of information, thereby achieving the effect of improving the accuracy of trajectory recovery. In addition, the method of realizing multimodal data fusion based on the second image is simple, does not require a complex fusion strategy, and has high applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is an application environment diagram of a vehicle trajectory recovery method based on unified imaging data modeling in one embodiment of the present application;

[0042] Figure 2 A schematic diagram of a process of a vehicle trajectory recovery method based on unified imaging data modeling in one embodiment of the present application;

[0043] Figure 3 A schematic diagram of a neural network training method in one embodiment of the present application;

[0044] Figure 4 A schematic diagram of a process of imaging trajectory recovery in one embodiment of the present application;

[0045] Figure 5 This is a schematic diagram of imaging expression and sequence expression in one embodiment of the present application;

[0046] Figure 6 A vehicle trajectory recovery device based on unified imaging data modeling in one embodiment of the present application;

[0047] Figure 7 This is a diagram of the internal structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] In the related art, in the field of trajectory recovery, trajectory recovery is usually achieved through clustering. One method is to construct an adjacency matrix based on vehicle driving probability, visual similarity and license plate similarity, and after integrating the snapshot identities across cameras through graph convolution, a clustering method is used to generate trajectory results; the adjacency matrix is ​​modified by iteratively estimating the trajectory probability, thereby improving the accuracy of trajectory recovery. Another method is to iteratively update the results through completion and denoising steps; wherein, in the denoising step, the normalized spatial coefficient is used to adjust the original snapshot similarity and filter out snapshots that violate the road network constraints. Furthermore, in the denoising step, the noise snapshots are removed by identifying a subset of high-probability clusters; and the clustering effect is improved by updating the dynamic features through the posterior probability of the path.

[0050] In the field of multi-target tracking, multiple detected targets are linked to continuous video frames in a single camera to achieve path trajectory generation; for multiple targets and multiple cameras, tracklets (target tracking track segments) of different cameras are connected through spatiotemporal cues to achieve path trajectory generation. For example, the tracks of adjacent cameras can be linked based on appearance and spatiotemporal similarity, and a robust tracklet representation can be generated using a spatiotemporal attention mechanism.

[0051] In the field of Map Matching, a method is proposed to use hidden Markov models combined with heuristic estimates of transition probabilities to infer the most likely trajectory. ST matching method: By utilizing spatial and temporal constraints, a candidate graph is constructed based on transition probabilities to identify the best matching path. IF matching method: By integrating speed, direction, and historical spatiotemporal context into transition probability estimates.

[0052] However, the related technologies have the following problems when realizing path trajectory generation: 1. Unable to make full use of road network data. The related methods failed to make full use of traffic network data, resulting in inaccurate trajectory inference results in complex or irregular road environments, which is more obvious in the recovery of low-precision positioning or non-continuous trajectories. 2. The spatial pattern of the trajectory sequence is unclear and the model generalization is insufficient. The spatial pattern in the trajectory sequence of the related methods is not clearly reflected. For example, adjacent trajectory points may not be connected in the sequence, which increases the difficulty of trajectory modeling. When trajectory recovery is achieved based on the application sequence model, the label bias and exposure bias problems further aggravate the instability of the model, resulting in the recovery effect depending on the data distribution and weak generalization. 3. The feature extraction and fusion design is complex and the effect depends on the strategy. When the related methods use road network data for modeling, they need to design specific feature extraction methods for different modal data. This mode significantly increases the complexity of model design. At the same time, the alignment and fusion effects of different modal features depend largely on the specific fusion strategy, and the model effect is easily affected by the strategy and unstable. 4. The existing expression methods of road network data have limitations, which make it impossible to fully express the complex spatiotemporal relationships required by trajectory sequences. For example, although the existing raster image modeling method can capture the mapping relationship, it is not good at expressing the global trajectory temporal relationship; the graph (image) expression form is good at expressing topological relationships, but it is weak in capturing the dynamic characteristics between trajectory points.

[0053] In view of the above problems, the vehicle trajectory recovery method based on unified imaging data modeling provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The data storage system can store the associated data required for executing the vehicle trajectory recovery method based on unified imaging data modeling, such as the road network and trajectory candidate points. Through the interaction between the terminal or the terminal and the server, the extraction and recovery of the road network trajectory are achieved. Among them, the terminal 102 can be but not limited to various personal computers, laptops, smart phones, tablet computers, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0054] In one embodiment, Figure 2 As shown in FIG, a vehicle trajectory recovery method based on unified imaging data modeling is provided. The method is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0055] Step S202: obtaining candidate points of the trajectory to be restored.

[0056] The candidate point is a known trajectory point in the trajectory to be restored. Affected by noise and trajectory positioning accuracy, the information contained in the candidate point may be biased. Optionally, the candidate point is obtained based on a snapshot query of the trajectory to be restored; and / or, the candidate point is obtained based on the positioning information of the trajectory to be restored in the positioning system.

[0057] Step S204, obtaining a first image according to the information of the road network where the trajectory to be restored is located.

[0058] The information of the road network is the road network structure and its related data. The first image is an image drawn based on the road network information. Optionally, the information of the road network includes the topological relationship of the road network, the physical characteristics of the road network including lanes, road surface types, road surface widths and other information, and the characteristics of the surrounding environment of the road. Optionally, the road network topological structure background is obtained, and the road network topological structure background is drawn as an image to obtain the first image.

[0059] Step S206, obtaining features of the candidate points on the first image according to the multimodal data of the candidate points, and combining the features with the first image to obtain a second image.

[0060] The candidate points of the trajectory to be restored may include multiple modal data, including but not limited to the time information when the candidate points are recorded, the location information such as the geographical location or relative geographical location of the candidate points, the motion state such as the moving direction and moving speed corresponding to the candidate points, the reliability of the candidate points, etc. The features of the candidate points on the first image include the image pixel color or grayscale value, the size of the candidate points in the first image, the location of the candidate points in the first image, etc.

[0061] Optionally, the features of the candidate point on the first image include the position, grayscale value and size of the pixel on the first image. The position of the pixel on the first image can be obtained based on the position information, the grayscale value of the candidate point on the first image can be obtained based on the time information, and the size of the candidate point on the first image can be obtained based on the reliability. It is understandable that the correspondence between various modal data and multiple features can be set according to application requirements. For example, the grayscale value of the candidate point can also be obtained based on the motion state or reliability, which will not be elaborated here.

[0062] Furthermore, before obtaining the features of the candidate points on the first image according to the multimodal data of the candidate points, the multimodal data of the plurality of candidate points may be processed so that the same type of data of the respective candidate points are at the same scale.

[0063] Step S208: obtaining a first trajectory prediction image of the trajectory to be restored according to the second image.

[0064] The trajectory prediction of the candidate points contained in the second image can be performed using a pre-trained model. The pre-trained model can be a machine learning model or a deep learning model. Optionally, the second image is input into a pre-trained deep neural network, and the deep neural network captures the constraint relationship between each node in the second image and the trajectory to be restored, thereby obtaining a first trajectory prediction image.

[0065] In the above-mentioned vehicle trajectory recovery method based on unified imaging data modeling, a second image is converted based on the first image containing the road network background and the candidate trajectory points. The second image simultaneously carries the location information, time information and topological structure information of the trajectory in the road network background, realizing the fusion of multimodal data, so that various types of information required for trajectory recovery can be extracted based on the second image, thereby improving the accuracy of trajectory recovery. The method of realizing multimodal data fusion based on the second image is simple. The unified image modality not only improves the information carrying capacity, but also breaks through the limitations of the traditional road network data expression method. It does not require complex fusion strategies and improves the applicability of the vehicle trajectory recovery method.

[0066] In one embodiment, image features of the candidate points on the first image are obtained based on multimodal data associated with the candidate points, including: mapping the candidate points to the first image based on the position information of the candidate points; obtaining the grayscale value of the candidate points in the first image based on the time information of the candidate points; and obtaining the range of pixels occupied by the candidate points in the first image based on the reliability of the candidate points.

[0067] Optionally, the position information of the candidate point is matched with the position in the first image. If the coordinate system of the candidate point is inconsistent with the coordinate system in the first image, the coordinate system of the first image or the candidate point may be transformed before mapping the candidate point.

[0068] Different time information corresponds to different pixel grayscale values. Optionally, the pixel grayscale value corresponding to the time information of the candidate point can be obtained by encoding, mapping the time information based on a preset mapping relationship, and other algorithms. The pixel grayscale value of the candidate point is obtained according to the time, and the difference between the recorded times of the candidate points can be intuitively reflected through the grayscale value in the second image.

[0069] The reliability of a candidate point refers to the accuracy and credibility of the motion state of a moving object in the trajectory to be restored relative to the actual motion state of the moving object. The reliability of a candidate point can be calculated based on the multimodal data of the candidate point. Optionally, the multimodal data of multiple candidate points on the trajectory to be restored are obtained, and the similarity between the multimodal data of the current candidate point and the multimodal data of other candidate points is compared. Based on the similarity, the behavior pattern of the moving object can be analyzed and understood, thereby judging the reliability of the current candidate point.

[0070] Furthermore, when the candidate point is obtained based on the positioning information of the trajectory to be restored in the positioning system, the positioning accuracy of the candidate point can be obtained based on the signal strength when the candidate point is collected and the accuracy of the device used to collect the candidate point: the higher the positioning accuracy, the higher the credibility; otherwise, the lower the credibility. The data integrity of the candidate point can also be obtained based on the data status such as whether the data is missing or repeated when the trajectory to be restored is collected: the more complete the data, the higher the credibility; otherwise, the lower the credibility.

[0071] When candidate points are obtained based on snapshot queries of the trajectory to be restored, the method for determining the reliability of the candidate points includes: clustering multiple snapshots of the trajectory to be restored based on visual methods such as extracting visual features, and calculating the similarity between the snapshots in each cluster and the cluster center. The higher the similarity, the higher the reliability of the candidate point corresponding to the snapshot; conversely, the lower the reliability of the candidate point. Each cluster obtained by clustering can be regarded as a candidate point sequence.

[0072] When candidate points are obtained based on the snapshot query of the trajectory to be restored, the method for determining the reliability of the candidate points may further include: directly performing target detection on the candidate points obtained based on the snapshot, synchronously obtaining the confidence of each candidate point, and using the confidence value as the reliability. Thus, the time of the snapshot, the position of the candidate point obtained by the camera, and the reliability of the candidate point are represented on the first image to obtain the second image.

[0073] The range of pixels occupied by the candidate point in the first image includes the shape and size of the candidate point in the first image. Optionally, the shape of the candidate point in the first image is set to a circle; the higher the reliability of the candidate point, the larger the range of pixels occupied in the first image; the lower the reliability of the candidate point, the smaller the range of pixels occupied in the first image. The shape and range of the candidate point in the first image can also be modified accordingly. The reliability of the candidate point can be intuitively presented through the pixel range in the second image, which can achieve the effect of strengthening the ability to identify key nodes in the trajectory prediction process.

[0074] In this embodiment, by acquiring the second image, multimodal data such as road network information, candidate point time, location, and reliability are fused, thereby converting the multimodal data into a unified imaging representation, thereby improving the fusion efficiency of the multimodal data while improving the multimodal expression capability of the data.

[0075] In one embodiment, the first network includes a first encoder and a decoder, and the second network includes a second encoder and a decoder. The first trajectory prediction image of the trajectory to be restored is predicted according to the second image, including: obtaining a third image generated according to the candidate points in the training trajectory, and a positive sample image and a negative sample image containing the training trajectory; using the third image as the input of the first network and the positive sample image as the supervision signal of the first network to train the first network; using the positive sample image as the input of the second network and the supervision signal of the second network to train the second network; training the first encoder and the second encoder according to the data output by the first encoder based on the third image and the data output by the second encoder based on the positive sample image and the negative sample image respectively; and obtaining the first trajectory prediction image predicted by the trained first network according to the second image.

[0076] Wherein, both the first network and the second network are deep learning neural networks. Optionally, in the first network, there are jump connections between some layers of the first encoder and the decoder. In the second network, there are jump connections between some layers of the second encoder and the decoder. Such an arrangement can prevent the gradient from disappearing during the learning process of the neural network.

[0077] The training trajectory is known data used to train the first encoder, the second encoder, and the decoder. The training trajectory can be constructed based on a real trajectory or extracted from an existing database.

[0078] The third image is generated based on the candidate points in the training trajectory. Optionally, the candidate points in the training trajectory and an image of the road network background containing the training trajectory are obtained, the corresponding grayscale values ​​are obtained based on the time information of the candidate points in the training trajectory, and the candidate points in the training trajectory are mapped to the image of the road network background containing the training trajectory based on the position information and grayscale values ​​of the candidate points in the training trajectory, so as to obtain the third image generated based on the candidate points in the training trajectory.

[0079] The positive sample image is an image obtained after the training trajectory is imaged, and the negative sample trajectory is constructed based on the positive sample image. Optionally, one or more known misconnection scenarios in the trajectory recovery scenario are obtained, and based on the training trajectory in the positive sample image, a trajectory connection path that conforms to the misconnection scenario is constructed, and the constructed trajectory connection path is imaged to obtain a negative sample image.

[0080] The supervisory signal is the target to be imitated during the supervised learning process of the first network and the second network. During the supervised training process, the first network and the second network are adjusted based on the supervisory signal as a standard so that the first network and the second network are as close to the corresponding supervisory signal as possible. Optionally, during the training of the first network, the third image is input into the first network, and the restored trajectory imaging representation is output. The positive sample image is used as the target of the restored trajectory imaging representation to adjust the parameters of the first network so that the output of the trained first network is close to the positive sample image. Optionally, during the training of the second network, the positive sample image is input into the second network, and the reconstructed trajectory imaging representation is output. The positive sample image is used as the target of the reconstructed trajectory imaging representation to adjust the parameters of the second network so that the output of the trained second network is close to the positive sample image.

[0081] Optionally, the second encoder outputs positive sample features and negative sample features based on the positive sample image and the negative sample image, respectively. The first encoder performs comparative training on the image features output by the third image, and adjusts the parameters of the first encoder and the second encoder through comparative training, so that the image features are as close to the positive sample features as possible while being away from the negative sample features.

[0082] In this embodiment, supervised learning is performed on the first encoder, the second encoder and the decoder to effectively capture the constraint relationship between each candidate point in the trajectory and the road network, thereby improving the accuracy of path recovery.

[0083] Furthermore, in one embodiment, the first network is trained using the third image as input and the positive sample image as a supervisory signal; the second network is trained using the positive sample image as input and supervisory signal; the first encoder and the second encoder are trained based on data output by the first encoder based on the third image and data output by the second encoder based on positive sample images and negative sample images, respectively, including: using the third image as input and the positive sample image as a supervisory signal to construct a first loss function for the first network; using the positive sample image as input and supervisory signal to construct a second loss function for the second network; constructing a third loss function based on data output by the first encoder based on the third image and data output by the second encoder based on positive sample images and negative sample images, respectively; and training the first encoder, the second encoder, and the decoder in combination with the first loss function, the second loss function, and the third loss function.

[0084] Among them, the first loss function is obtained according to the output of the first network and the positive sample image. The second loss function is obtained according to the output of the second network and the positive sample image. The third loss function is a contrast loss function, which is obtained according to the output of the first encoder, the positive sample features output by the second encoder based on the positive sample image, and the negative sample features output by the second encoder based on the negative sample image. Optionally, in order to improve the stability of contrast loss training and optimize the contrast training process, the first encoder is connected to the projection head, and the image features of the third image are output based on the projection head; the second encoder is connected to the projection head, and the positive sample features and negative sample features are output based on the projection head. Optionally, during the network training process, the first loss function, the second loss function and the third loss function can be obtained alternately to achieve alternating training of the first encoder, the second encoder and the decoder.

[0085] In this embodiment, multiple loss functions are used to implement the training of the first network, which can improve the generalization ability of the neural network.

[0086] Furthermore, in one embodiment, the first encoder, the second encoder and the decoder are trained in combination with the first loss function, the second loss function and the third loss function, including: constructing a comprehensive loss value according to the first loss function, the second loss function and the third loss function; when the comprehensive loss value satisfies the convergence condition, ending the training of the first encoder, the second encoder and the decoder.

[0087] The convergence condition may be that the comprehensive loss value reaches the minimum value, or that the number of iterations of one or more of the first network, the second network, the first encoder, and the second encoder reaches a specified number of times, or that the total number of iterations of the first network, the second network, the first encoder, and the second encoder reaches a specified number of times. Optionally, the first loss function, the second loss function, and the third loss function may be integrated by data methods such as addition and weighted summation to obtain a comprehensive loss value.

[0088] In this embodiment, by constructing a comprehensive loss value, the model is allowed to optimize its own performance from different angles, which can improve the robustness of the neural network.

[0089] In one embodiment, the grayscale value of the pixel corresponding to the candidate point in the first image is obtained according to the time information of the candidate point, including: obtaining multiple time information according to the timestamps of multiple candidate points; normalizing the multiple time information; encoding the normalized time information to obtain the pixel grayscale value of the candidate point.

[0090] Among them, the time information such as the year, month, day, and second when the candidate point is recorded can be obtained according to the timestamp. Optionally, the time information is obtained after the original information in the timestamp is converted into a numerical representation. The method for normalizing the time information includes: performing a linear transformation on the time information so that the time information is scaled to a specified range. For example, the time information can be normalized by Min-Max:

[0091]

[0092] You can also perform maximum absolute value scaling on the time information to normalize the time information:

[0093]

[0094] Among them, x max is the maximum time value in the time information, x min is the maximum time value in the time information, x is the time value currently being processed, and x' is the time value after processing.

[0095] Subsequently, based on a preset encoding method, the normalized time information is encoded so that the encoded value is within a preset grayscale value range. Furthermore, during encoding, the corresponding value of the time information can be encoded directly; or different components of the time information, such as year, month, day, hour, minute, and second, can be extracted and encoded as independent features to obtain the grayscale value of the pixel.

[0096] In this embodiment, by encoding the normalized timestamp, the original event information is converted into the grayscale value of the pixel, so that the time information of the candidate points can be integrated into the first image to capture the dynamic features between the candidate points.

[0097] In one embodiment, after obtaining the first trajectory prediction image of the trajectory to be restored according to the second image, the method further comprises: obtaining a road network trajectory sequence of the vehicle according to the degree of overlap between the first trajectory prediction image and the topological structure of the road network.

[0098] The first trajectory prediction image and the overlap of each node in the road network topology structure can be compared. Optionally, when the overlap is greater than or equal to a specified value, the node corresponding sequence is taken as part of the vehicle's road network trajectory sequence; when the overlap is less than the specified value, the node corresponding sequence is not taken as part of the vehicle's road network trajectory sequence.

[0099] Optionally, before the overlap between the first trajectory prediction image and the topological structure of the road network is predicted, the first image containing the road network topological structure is threshold segmented to obtain each node in the first image. Each node includes but is not limited to traffic structures such as road sections and intersections in the road network. By comparing the overlap between each node in the first image and the first trajectory prediction image, a road network trajectory sequence is obtained.

[0100] In this embodiment, by comparing the first trajectory prediction image with the topological structure of the road network, a road network trajectory sequence contained in the first trajectory prediction image is obtained based on the spatial constraints of the road network, thereby improving the accuracy of vehicle trajectory recovery.

[0101] In one embodiment, after predicting the first image of the trajectory to be restored according to the second image, the method further includes: acquiring a reference image of the trajectory to be restored; and evaluating the first image according to an image difference between the reference image and the first image.

[0102] The reference image is the real trajectory image of the trajectory to be restored, or it can be a more accurate trajectory image predicted based on other methods. The image difference includes but is not limited to the signal-to-noise ratio difference, structural similarity, and feature matching between the two images. The greater the image difference between the third image and the first trajectory prediction image, the lower the prediction accuracy; the smaller the image difference between the third image and the first trajectory prediction image, the higher the prediction accuracy. Optionally, the difference between the reference image and the first image is calculated by an image evaluation method, such as PSNR (Peak Signal-to-Noise Ratio) and MSE (Mean Squared Error), so as to achieve the evaluation of the first image.

[0103] In this embodiment, the evaluation result can be obtained through the similarity between adjacent nodes of the trajectory image, so that the importance of the node is taken into account in the evaluation process, which makes up for the problem that traditional indicators ignore important nodes of the trajectory and improves the accuracy of the evaluation.

[0104] In one embodiment, a trajectory recovery algorithm framework based on image representation is proposed, which supports multiple tasks such as trajectory recovery, multi-target tracking, and Map Matching, and realizes the extraction of road network constraint trajectories. Through the unified trajectory expression framework in time and space, combined with the deep learning model, high-precision and robust trajectory prediction is achieved. The implementation of the trajectory recovery algorithm framework is explained below using trajectory recovery as an example.

[0105] Establish a trajectory representation framework, and realize the imaging representation of candidate points through the trajectory representation framework. Optionally, obtain the road network topology background, draw the road network topology background as an image, and obtain the first image. Obtain the candidate point, and obtain the location information, time information and credibility of the candidate point, wherein the location information is the geographic coordinates of the candidate point, and the time information is the timestamp of the candidate point. According to the geographic coordinates of the candidate point, the candidate point is projected to the position corresponding to the first image; the timestamp is normalized and encoded to obtain the pixel grayscale value of each candidate point; the credibility of the candidate point is obtained according to the similarity between the current candidate point and other candidate points, and the size of the candidate point on the first image is obtained based on the credibility. Combine the position, grayscale value and size of the candidate point on the first image to obtain the second image, and realize the imaging representation of the candidate point. With this setting, while adding the time dimension characteristics to each candidate point, the information contained in the candidate point can be intuitively presented through the size and color of the pixel points in the image, thereby enhancing the recognition ability of key nodes in the trajectory prediction process. And based on the first image, a trajectory image is obtained after the trajectory to be restored is presented by specifying pixel values, and the road network topology structure contained in the first image provides accurate spatial constraints for trajectory recovery.

[0106] The trajectory sequence expression in related technologies cannot take into account the fusion of multimodal data such as time, space and road network at the same time, and the information carrying efficiency is low. Compared with related technologies, by converting the road network background and trajectory candidate points into a unified image representation, the image simultaneously carries the spatial coordinates, timestamp and road network background information of the trajectory, realizing the natural fusion of multimodal data including time, space and road network structure, which can significantly improve the multimodal expression ability of data and the efficiency of multimodal data fusion. At the same time, the fusion of multimodal data can also provide a unified input format for deep learning models.

[0107] Establish a trajectory recovery network. The trajectory recovery network can be used to achieve trajectory prediction based on the imaging representation of candidate points. Optionally, a deep neural network with an Encoder1 (first encoder) and a Decoder (decoder) structure is used as the network for prediction, that is, the above-mentioned first network. The second image is input to the first network to obtain the trajectory prediction graph output by the prediction module. Among them, the first network optimizes the MSE loss based on the real trajectory through supervised learning, captures the constraint relationship between the node and the road network trajectory, and achieves trajectory prediction.

[0108] The following is an explanation of the training method of the network used for prediction: obtain an imaging trajectory GT, that is, the positive sample image in the above embodiment. Obtain the candidate points contained in the training trajectory in the imaging trajectory GT, perform imaging processing on the candidate points in the training trajectory, and obtain an imaging representation of the candidate points, that is, the third image in the above embodiment. Considering that traditional trajectory prediction is prone to generate erroneous connection paths, such as: misconnection caused by noise nodes, trajectory breakage caused by node loss, and false connection between correct nodes. According to the trajectory in the positive sample image, simulate the above three erroneous connection scenarios to achieve negative sample trajectory generation, that is, construct a negative sample image. Figure 3 A neural network training method is provided, such as Figure 3 As shown, the first network includes Encoder1 (first encoder) and Decoder (decoder), and some levels of Encoder1 and Decoder are skipped. The second network includes Encoder2 (second encoder) and Decoder (decoder), and some levels of Encoder2 and Decoder are skipped. The decoders between the first network and the second network share parameters. The output end of the first encoder is connected to a projection head. The output end of the second encoder is connected to a projection head. The network training loss consists of three parts: the imaging expression of the candidate point is used as the input of the first network, the imaging trajectory GT is used as the supervision signal, and the first network is trained based on the MSE loss supervision. The GT trajectory map is used as the input of the second network and the supervision signal of the second network, and the second network is trained based on the reconstruction loss supervision.

[0109] The imaging trajectory GT and the negative sample image are respectively input into the second encoder, and the projection head connected to the second encoder outputs the positive sample features and the negative sample features. The image obtained by imaging the candidate points is input into the first encoder, and the input image features are obtained by the projection head connected to the first encoder. Encoder1 and Encoder2 are trained using infoNCE loss (contrastive learning loss). Based on the preset weights, the MSE loss, reconstruction loss, and contrast loss are weighted and summed to obtain a comprehensive loss value. When the comprehensive loss value meets the convergence condition, the training is completed to obtain the first network.

[0110] Based on the first network after training, Figure 4 An imaging trajectory recovery process is provided. Figure 4As shown, a road network background including a road network topology structure is obtained according to the road network data, that is, the first image in the above embodiment. Candidate points are obtained through the GPS positioning information or snapshot sequence of the trajectory to be restored. The imaging expression of the candidate points is realized by combining the road network background and the multimodal data of the candidate points. The first network after training performs network reasoning according to the image obtained by the imaging expression of the candidate points to obtain a trajectory recovery network, and then obtains the imaging expression of the restored trajectory, that is, the first trajectory prediction image in the above embodiment. The road network is threshold segmented to obtain each node of the road network. Among them, each node of the road network includes road surface information such as road sections and intersections. The road network trajectory sequence expression is obtained according to the overlap between the trajectory imaging expression and the road network node. Optionally, when the overlap between the trajectory imaging expression and the road network node is greater than or equal to a specified value, this road network node is used as data in the road network trajectory sequence; otherwise, this road network node is not used as data in the road network trajectory sequence. Figure 5 Schematic diagram of imaging expression and sequence expression in this embodiment. Figure 5 (a) is the imaging representation of the candidate points. Figure 5 (b) is the trajectory sequence expression, Figure 5 (c) is the trajectory imaging expression, Figure 5 (d) is the road network trajectory sequence expression. Figure 5 It can be seen that compared with sequence expression, imaging expression can achieve natural fusion of multimodal data.

[0111] After obtaining the graphical expression of the trajectory, the trajectory recovery result can be evaluated. Optionally, the real trajectory of the trajectory to be restored is obtained, and the real trajectory is expressed as an image to obtain a reference image. The difference between the reference image and the first trajectory prediction image is calculated by an image evaluation method to evaluate the accuracy of the predicted trajectory map. Since there are two adjacent nodes far apart in the first trajectory prediction image, if the sub-trajectory between the two nodes accounts for a longer total trajectory length, then the two nodes are relatively more important to the trajectory; if the sub-trajectory between the two nodes accounts for a shorter total trajectory length, then the two nodes are relatively less important to the trajectory, that is, the importance of the node can be reflected by the length information in the image. Therefore, based on the image evaluation of the differences between the image trajectories, the evaluation results obtained include considerations of the importance of the nodes, which can make up for the problem that traditional indicators ignore important nodes of the trajectory and provide more scientific and comprehensive evaluation results.

[0112] This embodiment uses image and gray value encoding to convert multimodal data into a unified imaging trajectory representation, realizing the fusion of multimodal data such as timestamps, spatial coordinates, and road network topology, and the fusion method is simple and highly practical. Based on a unified image representation, only one deep learning model is needed to capture the high-dimensional correlation features between modalities, significantly improving the accuracy of trajectory recovery. In addition, the deep learning model is obtained by comparative learning and training based on negative sample data and positive sample data. The model can synchronously learn the generation rules and noise processing capabilities of the trajectory, reduce the impact of noise candidate points and missing key nodes on the prediction results, and make the recovered trajectory close to the real trajectory, thereby improving the prediction accuracy.

[0113] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0114] Based on the same inventive concept, the embodiment of the present application also provides a vehicle trajectory recovery device based on unified imaging data modeling for implementing the vehicle trajectory recovery method based on unified imaging data modeling involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the vehicle trajectory recovery device based on unified imaging data modeling provided below can refer to the limitations of the vehicle trajectory recovery method based on unified imaging data modeling above, and will not be repeated here.

[0115] In one embodiment, Figure 6 As shown, a vehicle trajectory recovery device based on unified imaging data modeling is provided, comprising:

[0116] An acquisition module, used to acquire candidate points of the trajectory to be restored and a first image, wherein the first image contains a topological structure of a road network where the trajectory to be restored is located;

[0117] A first image construction module, used to obtain a first image according to information of a road network where the trajectory to be restored is located;

[0118] A second image construction module, used to obtain features of the candidate points on the first image according to the multimodal data of the candidate points, and combine the features with the first image to obtain a second image;

[0119] The prediction module is used to predict a first trajectory prediction image of the trajectory to be restored according to the second image.

[0120] In some of the embodiments, the second image construction module obtains features of the candidate points on the first image based on the multimodal data of the candidate points, and combines the features with the first image to obtain the second image, including: mapping the candidate points to the first image based on the position information of the candidate points; obtaining the grayscale value of the candidate points in the first image based on the time information of the candidate points; and obtaining the range of pixels occupied by the candidate points in the first image based on the reliability of the candidate points.

[0121] In some embodiments, the first network includes a first encoder and a decoder, the second network includes a second encoder and a decoder, and the prediction module predicts a first trajectory prediction image of the trajectory to be restored based on the second image, including: obtaining a third image generated based on candidate points in the training trajectory, a positive sample image and a negative sample image containing the training trajectory; using the third image as the input of the first network and the positive sample image as the supervision signal of the first network to train the first network; using the positive sample image as the input of the second network and the supervision signal of the second network to train the second network; training the first encoder and the second encoder based on data output by the first encoder based on the third image and data output by the second encoder based on the positive sample image and the negative sample image respectively; obtaining the first trajectory prediction image predicted by the trained first network based on the second image.

[0122] Optionally, the first network is trained using the third image as input and the positive sample image as a supervisory signal; the second network is trained using the positive sample image as input and supervisory signal; the first encoder and the second encoder are trained based on data output by the first encoder based on the third image and data output by the second encoder based on positive sample images and negative sample images, respectively, including: constructing a first loss function for the first network using the third image as input and the positive sample image as a supervisory signal; constructing a second loss function for the second network using the positive sample image as input and supervisory signal; constructing a third loss function based on data output by the first encoder based on the third image and data output by the second encoder based on positive sample images and negative sample images, respectively; and training the first encoder, the second encoder and the decoder in combination with the first loss function, the second loss function and the third loss function.

[0123] Optionally, the first encoder, the second encoder and the decoder are trained in combination with the first loss function, the second loss function and the third loss function, including: constructing a comprehensive loss value according to the first loss function, the second loss function and the third loss function; when the comprehensive loss value meets the convergence condition, ending the training of the first encoder, the second encoder and the decoder.

[0124] In some of the embodiments, the construction module obtains the grayscale value of the pixel corresponding to the candidate point in the first image based on the time information of the candidate point, including: obtaining multiple time information based on the timestamps of multiple candidate points; normalizing the multiple time information; encoding the normalized time information to obtain the pixel grayscale value of the candidate point.

[0125] In some of the embodiments, after the construction module predicts the first trajectory prediction image of the trajectory to be restored according to the second image, the execution method further includes: obtaining a road network trajectory sequence of the vehicle according to the overlap between the first trajectory prediction image and the topological structure of the road network.

[0126] In some of the embodiments, the vehicle trajectory recovery device based on unified imaging data modeling also includes an evaluation module, which is used to predict a first image of the trajectory to be recovered based on the second image. The method also includes: obtaining a reference image of the trajectory to be recovered; and evaluating the first image based on the image difference between the reference image and the first image.

[0127] Each module in the above-mentioned vehicle trajectory recovery device based on unified imaging data modeling can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0128] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a trajectory to be predicted, a first image, and a second image. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a vehicle trajectory recovery method based on unified imaging data modeling is implemented.

[0129] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0130] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0131] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0132] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0134] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A vehicle trajectory recovery method based on unified imaging data modeling, characterized in that: The method comprises: Obtain candidate points of the trajectory to be restored; Obtaining a first image according to information of a road network where the trajectory to be restored is located; Obtaining features of the candidate points on the first image according to the multimodal data of the candidate points, and combining the features with the first image to obtain a second image; A first trajectory prediction image of the trajectory to be restored is obtained according to the second image.

2. The method according to claim 1, characterized in that The obtaining a feature of the candidate point on the first image according to the multimodal data of the candidate point, and combining the feature with the first image to obtain a second image includes: Mapping the candidate point to the first image according to the position information of the candidate point; Obtaining a grayscale value of the candidate point in the first image according to the time information of the candidate point; The range of pixels occupied by the candidate point in the first image is obtained according to the reliability of the candidate point.

3. The method according to claim 1, characterized in that The first network includes a first encoder and a decoder, the second network includes a second encoder and the decoder, and the first trajectory prediction image of the trajectory to be restored is obtained by predicting the second image, including: Acquire a third image generated according to the candidate points in the training trajectory, a positive sample image and a negative sample image containing the training trajectory; Using the third image as an input of the first network and using the positive sample image as a supervision signal of the first network to train the first network; Using the positive sample image as the input of the second network and the supervision signal of the second network, training the second network; training the first encoder and the second encoder according to data output by the first encoder based on the third image and data output by the second encoder based on the positive sample image and the negative sample image respectively; The first trajectory prediction image is obtained by predicting the first trained network according to the second image.

4. The method according to claim 3, characterized in that Using the third image as input and the positive sample image as a supervisory signal, the first network is trained; Using the positive sample image as input and supervisory signal, the second network is trained; Training the first encoder and the second encoder according to the data output by the first encoder based on the third image and the data output by the second encoder based on the positive sample image and the negative sample image respectively, comprising: Taking the third image as input and the positive sample image as a supervision signal, constructing a first loss function of the first network; Taking the positive sample image as input and supervision signal, constructing a second loss function of the second network; constructing a third loss function according to data output by the first encoder based on the third image and data output by the second encoder based on the positive sample image and the negative sample image respectively; The first encoder, the second encoder, and the decoder are trained in combination with the first loss function, the second loss function, and the third loss function.

5. The method according to claim 4, characterized in that In combination with the first loss function, the second loss function and the third loss function, training the first encoder, the second encoder and the decoder comprises: Constructing a comprehensive loss value according to the first loss function, the second loss function and the third loss function; When the comprehensive loss value satisfies the convergence condition, the training of the first encoder, the second encoder and the decoder is terminated.

6. The method according to claim 1, characterized in that The obtaining, according to the time information of the candidate point, the grayscale value of the pixel corresponding to the candidate point in the first image includes: Obtaining a plurality of time information according to the timestamps of the plurality of candidate points; Normalizing the plurality of time information; The normalized time information is encoded to obtain the pixel grayscale value of the candidate point.

7. The method according to claim 1, characterized in that After obtaining the first trajectory prediction image of the trajectory to be restored according to the second image, the method further includes: A road network trajectory sequence of the vehicle is obtained according to the degree of overlap between the first trajectory prediction image and the topological structure of the road network.

8. The method according to claim 1, characterized in that After predicting the first image of the trajectory to be restored according to the second image, the method further includes: Acquire a reference image of the trajectory to be restored; The first image is evaluated based on an image difference between the reference image and the first image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images