Multimodal spatio-temporal trajectory fusion method, apparatus, device, medium and program product
By constructing a heterogeneous connectivity graph and using a graph convolutional neural network to generate embedded vectors, a multimodal spatiotemporal trajectory fusion method is proposed. This method solves the problem of poor fusion effect of multimodal trajectory data in sparse matrices, and achieves efficient and accurate trajectory fusion, adapting to data conditions with different characteristics.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN QIANHAI ZHONGDIAN HUIAN TECH CO LTD
- Filing Date
- 2026-01-13
- Publication Date
- 2026-05-12
AI Technical Summary
Existing trajectory fusion methods are ineffective in handling multimodal trajectory data with different characteristics, especially in sparse matrices, and have excessive computational burden, failing to meet real-time requirements.
A multimodal spatiotemporal trajectory fusion method is adopted. By acquiring multimodal spatiotemporal trajectory data to be processed, co-occurrence relationship pairs are determined based on preset time and space thresholds, a heterogeneous connectivity graph is constructed, and target embedding vectors are generated using graph convolutional neural networks. Finally, clustering and fusion are performed to achieve efficient and accurate trajectory fusion.
It achieves rapid and accurate trajectory fusion under big data conditions, adapts to multimodal and sparse data, reduces R&D and application costs, and improves the efficiency and accuracy of trajectory fusion.
Smart Images
Figure CN121502696B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of big data processing technology, and in particular to a multimodal spatiotemporal trajectory fusion method, apparatus, device, medium and program product. Background Technology
[0002] With the acceleration of urbanization, urban safety issues are receiving increasing attention. Smart and safe cities require the comprehensive application of various technologies, including information fusion, the Internet of Things (IoT), and artificial intelligence, to achieve information sharing and collaborative operation among different urban subsystems, thereby improving the city's safety and intelligence levels. Trajectory fusion technology, as an information fusion technology, can make significant contributions to smart and safe cities. Trajectory fusion technology is a technique that combines target trajectories obtained from different sensors or the same sensor at different time periods. By fusing different target trajectories, precise tracking and identification of targets can be achieved, improving the efficiency and accuracy of urban safety management. For example, it can improve the safety and management efficiency of public transportation, enhance urban security management efficiency, and improve urban emergency response efficiency, among other things.
[0003] In practical applications, the real-time requirements for trajectory fusion are extremely high for real-time monitoring and data analysis. Simultaneously, multiple trajectories with completely different modalities and characteristics often exist, originating from different sensors with varying sensor types, accuracies, and sampling rates. For example, surveillance cameras and ETC devices offer high positioning accuracy but have limited acquisition ranges, and surveillance cameras are prone to misidentification, while code-detecting devices have large acquisition ranges but low positioning accuracy. Trajectory fusion must consider the characteristics of these different modalities. Furthermore, due to limitations in the number and accuracy of sensors, there may be sparse matrices that are difficult to process. These sparse matrices can significantly impact the fused trajectory, and some important trajectory data may be hidden within them.
[0004] Existing trajectory fusion methods mainly include distance-based methods, time-series-based methods, and filtering-based methods. Distance-based methods first define a distance between trajectories, with closer distances indicating higher correlation. Then, they fuse the most relevant trajectories based on distance retrieval. The effectiveness of this type of method depends entirely on whether the distance definition matches the problem being solved and the data. Different modalities and data with different characteristics have different optimal trajectory distance definitions; therefore, this method performs poorly when fusing trajectories from different modalities with significantly different characteristics. Time-series-based methods first extract certain features based on time series data, and then use a model to determine whether trajectories should be fused based on these features. This type of method requires manually designed features and is highly dependent on temporal characteristics, making it difficult to handle the sparsity of temporal characteristics caused by different sampling rates. Furthermore, manually designed features are highly dependent on the designer's experience, resulting in poor performance. Filtering-based methods fuse observed data with prior knowledge to estimate the trajectory state. These methods mainly include Kalman filtering and particle filtering. Kalman filtering is suitable for linear systems, but its application in nonlinear systems is more difficult. In cities, the behavior patterns of people and vehicles are usually not simple linear laws. Particle filtering can be applied to nonlinear systems and systems with severe noise, but it is prone to excessive computational burden in large-scale systems. Summary of the Invention
[0005] This invention provides a multimodal spatiotemporal trajectory fusion method, apparatus, device, medium, and program product to achieve rapid and accurate trajectory fusion under big data conditions, and to adapt well to multimodal and sparse data.
[0006] In a first aspect, embodiments of the present invention provide a multimodal spatiotemporal trajectory fusion method, the method comprising:
[0007] Acquire multimodal spatiotemporal trajectory data to be processed;
[0008] Based on preset time thresholds and preset space thresholds, the spatiotemporal co-occurrence pairs of the spatiotemporal trajectory data to be processed in each modality are determined.
[0009] Edges are established based on the co-occurrence pairs to construct a heterogeneous connection graph between nodes of each modality. Each edge carries attributes including the spatiotemporal co-occurrence features and association strength features of the corresponding co-occurrence pairs.
[0010] The heterogeneous connection graph is input into a trained graph convolutional neural network to generate the target embedding vector;
[0011] The target embedding vectors are clustered and fused according to a preset similarity threshold to obtain the trajectory fusion result.
[0012] Optionally, before establishing edges based on the co-occurrence relationships to construct a heterogeneous connection graph between modal nodes, the method further includes:
[0013] The number of co-occurrence locations and the total number of co-occurrences for each co-occurrence pair are used as the spatiotemporal co-occurrence feature.
[0014] Optionally, the association strength feature includes an association strength score; before establishing edges based on the co-occurrence relationship to construct a heterogeneous connection graph between modal nodes, the method further includes:
[0015] Statistically analyze the co-occurrence event segments that occur consecutively on the timeline for each of the aforementioned co-occurrence pairs;
[0016] The association strength score of the corresponding co-occurrence relationship pair is determined based on the number of times each co-occurrence event fragment appears and the duration of the fragment.
[0017] Optionally, the association strength feature may also include point-to-point mutual information for the corresponding co-occurrence pairs.
[0018] Optionally, determining the spatiotemporally adjacent co-occurrence pairs between the spatiotemporal trajectory data to be processed for each modality based on preset time thresholds and preset spatial thresholds includes:
[0019] The preset time threshold and the preset space threshold are dynamically adjusted based on the target movement speed.
[0020] Optionally, before inputting the heterogeneous connection graph into the trained graph convolutional neural network to generate the target embedding vector, the method further includes:
[0021] Acquire multimodal sample spatiotemporal trajectory data and sample relationship pair annotation data;
[0022] Construct a heterogeneous connection graph of samples based on the spatiotemporal trajectory data of the samples;
[0023] The heterogeneous connectivity graph of the samples is used as the model input, and the labeled data of the sample relationships is used as the supervision signal. A semi-supervised learning algorithm is used to train the graph convolutional neural network.
[0024] Secondly, embodiments of the present invention also provide a multimodal spatiotemporal trajectory fusion device, the device comprising:
[0025] The data acquisition module is used to acquire multimodal spatiotemporal trajectory data to be processed.
[0026] The relationship pair determination module is used to determine the spatiotemporally adjacent co-occurrence relationship pairs between the spatiotemporal trajectory data to be processed in each modality based on preset time thresholds and preset spatial thresholds.
[0027] The connection graph construction module is used to establish edges based on the co-occurrence relationship pairs to construct a heterogeneous connection graph between each modality node. Each edge carries attributes including the spatiotemporal co-occurrence characteristics and association strength characteristics of the corresponding co-occurrence relationship pairs.
[0028] An embedding vector generation module is used to input the heterogeneous connection graph into a trained graph convolutional neural network to generate a target embedding vector.
[0029] The trajectory fusion module is used to cluster and fuse the target embedding vectors according to a preset similarity threshold to obtain the trajectory fusion result.
[0030] Thirdly, embodiments of the present invention also provide a computer device, the computer device comprising:
[0031] One or more processors;
[0032] Memory, used to store one or more programs;
[0033] When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal spatiotemporal trajectory fusion method provided in any embodiment of the present invention.
[0034] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multimodal spatiotemporal trajectory fusion method provided in any embodiment of the present invention.
[0035] Fifthly, embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the multimodal spatiotemporal trajectory fusion method provided in any embodiment of the present invention.
[0036] This invention provides a multimodal spatiotemporal trajectory fusion method. First, it acquires multimodal spatiotemporal trajectory data to be processed. Then, based on preset time and spatial thresholds, it determines spatiotemporally proximate co-occurrence pairs between the spatiotemporal trajectory data of each modality. Next, it establishes edges based on these co-occurrence pairs to construct a heterogeneous connectivity graph between nodes of each modality. The spatiotemporal co-occurrence features and association strength features of the co-occurrence pairs are used as attributes carried by the corresponding edges. Subsequently, the obtained heterogeneous connectivity graph is input into a trained graph convolutional neural network to generate target embedding vectors. Finally, it performs clustering and fusion on the obtained target embedding vectors according to a preset similarity threshold to obtain the trajectory fusion result. The multimodal spatiotemporal trajectory fusion method provided by this invention maps spatiotemporal trajectories to a spatiotemporal topology graph, enabling graph convolutional neural networks to be successfully applied in the field of trajectory fusion, thereby achieving efficient and accurate trajectory fusion. This approach also enables data from different modalities to be represented in the same way, bridging the gap between modalities with varying characteristics. It eliminates the need to design separate algorithms and train models for each modality, significantly reducing R&D, application costs, and improving efficiency. Furthermore, it allows for easy and rapid expansion across multimodal trajectory data with different characteristics. Additionally, the graph structure transformation of sparse data yields more global spatiotemporal correlation information, making the data representation less sparse and avoiding interference from sparse data on the model. This allows the model to adapt to different data conditions, such as dense and sparse trajectories. Attached Figure Description
[0037] Figure 1 This is a flowchart of the multimodal spatiotemporal trajectory fusion method provided in Embodiment 1 of the present invention;
[0038] Figure 2 This is an exemplary heterogeneous connection diagram provided in Embodiment 1 of the present invention;
[0039] Figure 3 This is a schematic diagram of the structure of the multimodal spatiotemporal trajectory fusion device provided in Embodiment 2 of the present invention;
[0040] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation
[0041] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.
[0042] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.
[0043] Example 1
[0044] Figure 1 This is a flowchart of a multimodal spatiotemporal trajectory fusion method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations requiring the fusion of spatiotemporal trajectories of multiple modalities in application scenarios such as smart city systems, urban intelligent security systems, image-code joint detection systems, and vehicle-code association systems. This method can be executed by the multimodal spatiotemporal trajectory fusion device provided in this embodiment of the invention. This device can be implemented in hardware and / or software, and is generally integrated into computer equipment. Figure 1 As shown, the method specifically includes the following steps:
[0045] S11. Obtain the multimodal spatiotemporal trajectory data to be processed.
[0046] S12. Based on preset time thresholds and preset space thresholds, determine the spatiotemporal co-occurrence pairs of the spatiotemporal trajectory data to be processed for each modality.
[0047] S13. Establish edges based on the co-occurrence pairs to construct a heterogeneous connection graph between each modality node. Each edge carries attributes including the spatiotemporal co-occurrence characteristics and association strength characteristics of the corresponding co-occurrence pairs.
[0048] S14. Input the heterogeneous connection graph into the trained graph convolutional neural network to generate the target embedding vector.
[0049] S15. Cluster and fuse the target embedding vectors according to a preset similarity threshold to obtain the trajectory fusion result.
[0050] Specifically, various data acquisition devices deployed on-site can collect the required spatiotemporal trajectory data of different modalities in real time. Different modal combinations can be collected for different scenarios, allowing for the fusion of trajectory data from various modalities as needed. For example, in a vehicle-to-code (V2C) association system, the spatiotemporal trajectory data to be processed may include IMSI data and license plate image data. IMSI data can be collected by a code detection device, which can collect IMSI signals within a certain range around it and record data such as latitude and longitude, time, IMSI number (International Mobile Subscriber Identity), and number location. Depending on the device type and parameter settings, the collection range typically ranges from 200 to 1500 meters. The IMSI number is used to distinguish different users on the mobile network; for example, each mobile phone SIM card has a unique IMSI number. Before the method is implemented, the detection equipment can be deployed and verified, such as configuring relevant parameters (frequency band, sensitivity, etc.) and conducting field tests and link verification to ensure that the equipment can continuously and stably capture the IMSI signal emitted by mobile devices and the spatiotemporal information it carries (such as timestamps, signal strength, associated base stations / locations, etc.). License plate image data can be collected through license plate cameras. Before the method is implemented, the license plate cameras can also be deployed and verified, such as optimizing the installation angle, lighting adaptability, and triggering mechanism (such as inductive loop or video stream triggering), as well as conducting system integration and image quality assessment to ensure that the license plate cameras can continuously and stably collect clear and identifiable license plate images with accurate spatiotemporal information (timestamps and location information, such as GPS coordinates). By pre-deploying and verifying the acquisition equipment, the continuity and stability of the original multimodal spatiotemporal trajectory data input can be ensured, thus providing a reliable foundation for subsequent processing.
[0051] After collecting various modalities of spatiotemporal trajectory data to be processed, the data can be preprocessed and a corresponding spatiotemporal database can be built for storage, providing a high-quality, quickly searchable data foundation for subsequent processing. As in the example above, for license plate image data, an OCR (Optical Character Recognition) system can be used to process the license plate images to automatically identify and extract the license plate number text information, thus transforming unstructured image data into structured text identifiers. Furthermore, IMSI data and the transformed license plate image data can be cleaned based on missing key fields (such as latitude and longitude, timestamps, etc.). Data lacking key fields cannot provide effective spatiotemporal positioning information and is not suitable for subsequent trajectory fusion calculations; removing these fields helps improve data quality and optimize storage efficiency. Subsequently, the preprocessed data can be persistently stored in the database, including the license plate number text information, IMSI number, and its corresponding spatiotemporal information (such as latitude and longitude, timestamps, etc.). To address subsequent high-frequency query needs, efficient indexes can be created in the database for core fields such as timestamps, latitude and longitude, and entity IDs (license plate numbers, IMSI numbers), thereby significantly improving the speed of data retrieval and retrieval, and providing support for real-time or near-real-time processing.
[0052] After obtaining the required spatiotemporal trajectory data to be processed, the spatiotemporal trajectory data stored in the aforementioned spatiotemporal database can be used to determine co-occurrence pairs between different modalities that are spatiotemporally adjacent, based on preset time and spatial thresholds. An example is the <license plate, IMSI> pair. Co-occurrence pairs can be determined according to the "simultaneous and same location" rule, such as similar timestamps (duration less than a preset time threshold) and similar latitude and longitude (distance less than a preset spatial threshold). Specifically, the spatiotemporal trajectory data of the target modality can be scanned to identify other modalities that are spatiotemporally adjacent to each data point, thereby generating corresponding co-occurrence pairs. For example, for each license plate image data, the IMSI data that is spatiotemporally adjacent to it can be identified.
[0053] In an optional implementation, determining the spatiotemporally adjacent co-occurrence pairs between the spatiotemporal trajectory data to be processed for each modality based on a preset time threshold and a preset spatial threshold includes: dynamically adjusting the preset time threshold and the preset spatial threshold according to the target moving speed.
[0054] Specifically, the target moving speed can be obtained by matching and measuring the spatiotemporal trajectory data of the target modality. For example, vehicle speed can be collected in real time as the target moving speed while acquiring license plate image data. For the preset time threshold, a fixed time window can be abandoned, adaptively decreasing when the target moving speed is high (e.g., greater than the preset speed threshold) to adapt to fast-moving scenarios, and increasing when the target moving speed is low (e.g., less than or equal to the preset speed threshold) to tolerate brief delays or interruptions in signal acquisition. For example, a shorter time window (e.g., 30 seconds) is used when the vehicle speed is greater than 30 km / h, while a longer time window (e.g., 5 minutes) is used when the vehicle speed is less than or equal to 30 km / h (including when stationary). For the preset spatial threshold, the higher the target moving speed, the higher the tolerance can be set. Specifically, a distance threshold (e.g., target moving speed × n, for example, n=2) can be determined in real time based on the target moving speed, and the larger value between this distance threshold and the default distance threshold is used as the preset spatial threshold. For example, the preset spatial threshold = max(50, vehicle speed × 2).
[0055] After obtaining the required co-occurrence pairs, a heterogeneous connectivity graph with attributes can be modeled based on the co-occurrence relationships to provide structured input data for subsequent graph convolutional neural networks. Specifically, a heterogeneous connectivity graph containing multiple types of nodes can be constructed, with each type of node corresponding to a modality, and edges can be established between co-occurrence pairs. For example, a heterogeneous connectivity graph constructed based on two types of nodes (license plate, IMSI) would look like this: Figure 2 As shown. Each edge carries attributes including the spatiotemporal co-occurrence features and association strength features of the corresponding co-occurrence relationship pair. Optionally, the spatiotemporal co-occurrence features may include the number of co-occurrence locations and / or the total number of co-occurrences, both of which are integers. The association strength features may include the association strength score and / or the mutual information between points, both of which are floating-point numbers.
[0056] In an optional implementation, before establishing edges based on the co-occurrence pairs to construct a heterogeneous connectivity graph between modal nodes, the method further includes: counting the number of co-occurrence locations and the total number of co-occurrences for each co-occurrence pair as the spatiotemporal co-occurrence feature. The number of co-occurrence locations refers to the number of unique geographical locations where co-occurrence events occur for the same co-occurrence pair, which can be pre-divided into geographical locations; the total number of co-occurrences refers to the total number of times the same co-occurrence pair occurs at different time points.
[0057] In an optional implementation, the association strength feature includes an association strength score; before establishing edges based on the co-occurrence pairs to construct a heterogeneous connection graph between modal nodes, the method further includes: counting co-occurrence event segments that occur consecutively on the timeline for each co-occurrence pair; and determining the association strength score of the corresponding co-occurrence pair based on the number of co-occurrences and the duration of each co-occurrence event segment.
[0058] Specifically, for the attributes carried by edges in a heterogeneous connectivity graph, we can not only rely on the total number of co-occurrences, but also emphasize the weight of consecutive co-occurrence event segments. For example, we can assign a higher weight to co-occurrence patterns that are accompanied continuously for a long time, i.e., the association strength score will be significantly better than short, discrete, accidental encounters. For instance, the score for license plate A and IMSI_X co-occurring five times consecutively between 10:00 and 10:15 is much higher than the score for five co-occurrences scattered at different time points. Specifically, we can first count the consecutive co-occurrence event segments that occur in each co-occurrence relationship pair on the timeline, and determine the number of co-occurrences and the duration of each co-occurrence event segment. Then, we can calculate the association strength score of the corresponding co-occurrence relationship pair based on the number of co-occurrences and the duration of each co-occurrence event segment. Specifically, we can assign a higher association strength score to co-occurrence relationship pairs with more co-occurrences and / or longer durations. Preferably, we can use the following formula to calculate the association strength score: Association Strength Score = ∑[Number of Co-occurrences within a Segment × log(Segment Duration + 1)], that is, summing all co-occurrence event segments for each co-occurrence relationship pair.
[0059] Optionally, the association strength feature further includes the point-to-point mutual information (PMI) of the corresponding co-occurrence pairs, thereby suppressing spurious association biases caused by high-frequency IMSI data or high-frequency license plate image data. A high PMI value indicates that the co-occurrence significantly exceeds random expectations and has a strong association, while a negative PMI value or close to zero indicates that the co-occurrence may be purely accidental or influenced by high-frequency individuals. The formula for calculating point-to-point mutual information is: PMI = log(Pc / (P1×P2)), where Pc represents the joint co-occurrence probability of the corresponding co-occurrence pair, which can be estimated based on the number of co-occurrence events, and P1 and P2 represent the marginal probabilities of the two elements in the corresponding co-occurrence pair occurring independently, which can be estimated based on the total number of their respective occurrence events. For example, for the co-occurrence relationship between license plate and IMSI, the calculation formula is: PMI(car,imsi)=log(P(car,imsi) / (P(car)×P(imsi))), where P(car,imsi) is the joint probability of co-occurrence of the corresponding license plate and IMSI, and P(car) and P(imsi) are the marginal probabilities of the corresponding license plate and IMSI occurring independently.
[0060] After constructing the heterogeneous connectivity graph, it can be input into a pre-trained graph convolutional neural network (GNN) to generate target embedding vectors representing trajectories. Graph Neural Networks (GNNs) are neural networks based on graph-structured data, greatly expanding the types of data that neural networks can apply. Graph Convolutional Neural Networks (GCNs) extend convolution to graph structures, enabling better local feature learning and abstraction through graph convolution computation. Traditional GNNs are commonly used in social network analysis, recommendation systems, and knowledge graphs. However, because spatiotemporal trajectory data is completely different from text and image data, and the data volume is extremely large, sparse trajectories may appear. Since sparse trajectories have weak temporal dependencies, existing solutions cannot directly apply GNNs to trajectory fusion problems. However, through the heterogeneous connectivity graph modeling method in this embodiment, GNNs can be effectively applied to solve the trajectory fusion problem.
[0061] After generating the target embedding vectors, density clustering algorithms (such as DBSCAN) or hierarchical clustering algorithms (such as HDBSCAN) can be used to perform cluster analysis on the obtained target embedding vectors. The clustering basis can be cosine similarity or Euclidean distance to measure the vector space relationship, etc. Through clustering, several clusters can be generated, each containing a set of highly correlated matching embedding vectors, such as license plate-IMSI embedding vectors. During the clustering process, an appropriate preset similarity threshold can be selected based on experimental and test results, so that the trajectories that meet the preset similarity threshold can be fused to obtain the trajectory fusion result. This achieves accurate fusion of multimodal trajectories and obtains the associated trajectories of multimodal devices (such as vehicles and mobile devices).
[0062] Based on the above technical solution, optionally, before inputting the heterogeneous connection graph into the trained graph convolutional neural network to generate the target embedding vector, the method further includes: acquiring multimodal sample spatiotemporal trajectory data and sample relationship pair annotation data; constructing a sample heterogeneous connection graph based on the sample spatiotemporal trajectory data; using the sample heterogeneous connection graph as model input and the sample relationship pair annotation data as supervision signals, and training the graph convolutional neural network using a semi-supervised learning algorithm.
[0063] Specifically, the type and acquisition method of the sample spatiotemporal trajectory data are similar to the spatiotemporal trajectory data to be processed mentioned above, and the same preprocessing procedure can be performed. Then, a similar approach can be used to determine the spatiotemporally adjacent sample co-occurrence pairs between different modalities in the sample spatiotemporal trajectory data, and dynamic spatiotemporal threshold rules can also be used in this process. Furthermore, a similar approach can be used to model an attributed sample heterogeneous connectivity graph based on the obtained sample co-occurrence pairs, thereby providing structured input data for graph convolutional neural network (GNN) learning, enabling it to output discriminative embedding vectors for trajectory similarity judgment. Subsequently, the obtained sample heterogeneous connectivity graph can be used as model input, and the pre-annotated sample relationship pair labeling data can be used as a supervision signal. A semi-supervised learning algorithm is then used to train the GNN, allowing it to learn to automatically mine deep spatiotemporal association patterns of multimodal data from the heterogeneous connectivity graph and achieve robust fusion of sparse trajectories. The sample relationship pair labeling data can include positive and negative sample pairs, i.e., associated entity pairs and non-associated entity pairs. For example, positive samples can be car-imsi association pairs of the same vehicle, and negative samples can be randomly sampled non-associated entity pairs. Once training is complete, the trained graph convolutional neural network can be deployed for the aforementioned practical application inference process. During the training phase, data from various modalities can be coupled together for training. After deployment, data from different modalities can be inferred pairwise to reduce computational complexity.
[0064] The technical solution provided by this invention first acquires multimodal spatiotemporal trajectory data to be processed. Then, based on preset time and spatial thresholds, it determines the spatiotemporally adjacent co-occurrence pairs between the spatiotemporal trajectory data of each modality. Next, it establishes edges based on the obtained co-occurrence pairs to construct a heterogeneous connection graph between nodes of each modality. The spatiotemporal co-occurrence features and association strength features of the co-occurrence pairs are used as attributes carried by the corresponding edges. Subsequently, the obtained heterogeneous connection graph is input into a trained graph convolutional neural network to generate target embedding vectors. Then, based on a preset similarity threshold, the obtained target embedding vectors are clustered and fused to obtain the trajectory fusion result. By mapping spatiotemporal trajectories to a spatiotemporal topology graph, graph convolutional neural networks can be successfully applied in the field of trajectory fusion, thereby achieving efficient and accurate trajectory fusion. Simultaneously, it enables data from different modalities to be represented in the same way, bridging the gap between modal data with different characteristics. It eliminates the need to design algorithms and train models separately for each modality, greatly saving R&D, application costs, and efficiency. Furthermore, it can be easily and quickly extended to multimodal trajectory data with different characteristics. In addition, sparse data is transformed through graph structure to obtain more global spatiotemporal correlation information, making the data representation no longer sparse, avoiding interference from sparse data on the model, and adapting to different data conditions such as dense trajectories and sparse trajectories.
[0065] Example 2
[0066] Figure 3 This is a schematic diagram of the multimodal spatiotemporal trajectory fusion device provided in Embodiment 2 of the present invention. This device can be implemented in hardware and / or software, and is generally integrated into a computer device to execute the multimodal spatiotemporal trajectory fusion method provided in any embodiment of the present invention. Figure 3 As shown, the device includes:
[0067] The data acquisition module 21 is used to acquire multimodal spatiotemporal trajectory data to be processed;
[0068] The relationship pair determination module 22 is used to determine the co-occurrence relationship pairs between the spatiotemporally adjacent data of the spatiotemporal trajectory data to be processed in each modality based on a preset time threshold and a preset space threshold.
[0069] The connection graph construction module 23 is used to establish edges based on the co-occurrence relationship pairs to construct a heterogeneous connection graph between each modal node. Each edge carries attributes including the spatiotemporal co-occurrence characteristics and association strength characteristics of the corresponding co-occurrence relationship pairs.
[0070] Embedding vector generation module 24 is used to input the heterogeneous connection graph into a trained graph convolutional neural network to generate a target embedding vector;
[0071] The trajectory fusion module 25 is used to cluster and fuse the target embedding vector according to a preset similarity threshold to obtain the trajectory fusion result.
[0072] The technical solution provided by this invention first acquires multimodal spatiotemporal trajectory data to be processed. Then, based on preset time and spatial thresholds, it determines the spatiotemporally adjacent co-occurrence pairs between the spatiotemporal trajectory data of each modality. Next, it establishes edges based on the obtained co-occurrence pairs to construct a heterogeneous connection graph between nodes of each modality. The spatiotemporal co-occurrence features and association strength features of the co-occurrence pairs are used as attributes carried by the corresponding edges. Subsequently, the obtained heterogeneous connection graph is input into a trained graph convolutional neural network to generate target embedding vectors. Then, based on a preset similarity threshold, the obtained target embedding vectors are clustered and fused to obtain the trajectory fusion result. By mapping spatiotemporal trajectories to a spatiotemporal topology graph, graph convolutional neural networks can be successfully applied in the field of trajectory fusion, thereby achieving efficient and accurate trajectory fusion. Simultaneously, it enables data from different modalities to be represented in the same way, bridging the gap between modal data with different characteristics. It eliminates the need to design algorithms and train models separately for each modality, greatly saving R&D, application costs, and efficiency. Furthermore, it can be easily and quickly extended to multimodal trajectory data with different characteristics. In addition, sparse data is transformed through graph structure to obtain more global spatiotemporal correlation information, making the data representation no longer sparse, avoiding interference from sparse data on the model, and adapting to different data conditions such as dense trajectories and sparse trajectories.
[0073] Based on the above technical solution, optionally, the device further includes:
[0074] The spatiotemporal co-occurrence feature statistics module is used to count the number of co-occurrence locations and the total number of co-occurrences of each co-occurrence relationship pair as the spatiotemporal co-occurrence feature before establishing edges based on the co-occurrence relationship pairs to construct a heterogeneous connection graph between each modal node.
[0075] Based on the above technical solution, optionally, the association strength feature includes an association strength score; the device further includes:
[0076] The co-occurrence event fragment statistics module is used to count the co-occurrence event fragments that occur continuously on the timeline for each co-occurrence relationship pair before establishing edges based on the co-occurrence relationship pairs to construct a heterogeneous connection graph between each modal node;
[0077] The association strength score determination module is used to determine the association strength score of the corresponding co-occurrence relationship pair based on the number of co-occurrences and the duration of each co-occurrence event fragment.
[0078] Based on the above technical solution, optionally, the association strength feature also includes the inter-point mutual information of the corresponding co-occurrence relationship pair.
[0079] Based on the above technical solution, optionally, the relationship pair determination module 22 includes:
[0080] The spatiotemporal threshold dynamic adjustment unit is used to dynamically adjust the preset time threshold and the preset spatial threshold according to the target's moving speed.
[0081] Based on the above technical solution, optionally, the device further includes:
[0082] The sample data acquisition module is used to acquire multimodal sample spatiotemporal trajectory data and sample relationship pair annotation data before inputting the heterogeneous connection graph into the trained graph convolutional neural network to generate the target embedding vector;
[0083] The sample connectivity graph construction module is used to construct a heterogeneous connectivity graph of samples based on the sample spatiotemporal trajectory data.
[0084] The network training module is used to train the graph convolutional neural network by taking the sample heterogeneous connection graph as model input and the sample relationship pair labeled data as supervision signal, and using a semi-supervised learning algorithm.
[0085] The multimodal spatiotemporal trajectory fusion device provided in the embodiments of the present invention can execute the multimodal spatiotemporal trajectory fusion method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0086] It is worth noting that in the embodiments of the multimodal spatiotemporal trajectory fusion device described above, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.
[0087] Example 3
[0088] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention, showing a block diagram of an exemplary computer device suitable for implementing the embodiments of the present invention. Figure 4 The computer device shown is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the present invention. Figure 4 As shown, the computer device includes a processor 31, a memory 32, an input device 33, and an output device 34; the number of processors 31 in the computer device can be one or more. Figure 4 Taking a processor 31 as an example, the processor 31, memory 32, input device 33, and output device 34 in a computer device can be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.
[0089] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the multimodal spatiotemporal trajectory fusion method in this embodiment of the invention (e.g., the data acquisition module 21, relation pair determination module 22, connection graph construction module 23, embedding vector generation module 24, and trajectory fusion module 25 in the multimodal spatiotemporal trajectory fusion device). The processor 31 executes various functional applications and data processing of the computer device by running the software programs, instructions, and modules stored in the memory 32, thereby realizing the aforementioned multimodal spatiotemporal trajectory fusion method.
[0090] The memory 32 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 32 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 32 may further include memory remotely located relative to the processor 31, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0091] The input device 33 can be used to acquire multimodal spatiotemporal trajectory data to be processed, and to generate key signal inputs related to user settings and function control of the computer device. The output device 34 may include a display screen, which can be used to display trajectory fusion results to the user, etc.
[0092] Example 4
[0093] Embodiment 4 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a multimodal spatiotemporal trajectory fusion method, the method comprising:
[0094] Acquire multimodal spatiotemporal trajectory data to be processed;
[0095] Based on preset time thresholds and preset space thresholds, the spatiotemporal co-occurrence pairs of the spatiotemporal trajectory data to be processed in each modality are determined.
[0096] Edges are established based on the co-occurrence pairs to construct a heterogeneous connection graph between nodes of each modality. Each edge carries attributes including the spatiotemporal co-occurrence features and association strength features of the corresponding co-occurrence pairs.
[0097] The heterogeneous connection graph is input into a trained graph convolutional neural network to generate the target embedding vector;
[0098] The target embedding vectors are clustered and fused according to a preset similarity threshold to obtain the trajectory fusion result.
[0099] Storage media can be any type of memory device or storage device. The term "storage media" is intended to include: mounting media, such as CD-ROMs, floppy disks, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disks or optical storage); registers or other similar types of memory elements. Storage media may also include other types of memory or combinations thereof. Furthermore, storage media may reside in a computer system in which the program is executed, or may reside in a different second computer system connected to the computer system via a network (such as the Internet). The second computer system can provide program instructions to the computer for execution. The term "storage media" can include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). Storage media may store program instructions (e.g., specifically implemented as a computer program) that can be executed by one or more processors.
[0100] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the method operations described above, but can also execute related operations in the multimodal spatiotemporal trajectory fusion method provided in any embodiment of the present invention.
[0101] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.
[0102] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0103] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0104] Example 5
[0105] Embodiment 5 of the present invention also provides a computer program product, which includes a computer program (also referred to as code or instructions). The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it is used to execute the multimodal spatiotemporal trajectory fusion method provided in any of the above embodiments, and has the corresponding beneficial effects of the execution method.
[0106] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.
Claims
1. A multimodal spatiotemporal trajectory fusion method, characterized in that, include: Acquire multimodal spatiotemporal trajectory data to be processed, the spatiotemporal trajectory data to be processed including IMSI data and license plate image data; Based on preset time thresholds and preset space thresholds, the spatiotemporal co-occurrence pairs of the spatiotemporal trajectory data to be processed in each modality are determined. Edges are established based on the co-occurrence pairs to construct a heterogeneous connection graph between nodes of each modality. Each edge carries attributes including the spatiotemporal co-occurrence features and association strength features of the corresponding co-occurrence pairs. The heterogeneous connection graph is input into a trained graph convolutional neural network to generate the target embedding vector; The target embedding vectors are clustered and fused according to a preset similarity threshold to obtain the trajectory fusion result; The association strength feature includes an association strength score; Before establishing edges based on the co-occurrence relationship to construct a heterogeneous connection graph between modal nodes, the method further includes: Statistically analyze the co-occurrence event segments that occur consecutively on the timeline for each of the aforementioned co-occurrence pairs; The association strength score of the corresponding co-occurrence relationship pair is determined based on the number of times each co-occurrence event fragment appears and the duration of the fragment.
2. The multimodal spatiotemporal trajectory fusion method according to claim 1, characterized in that, Before establishing edges based on the co-occurrence relationship to construct a heterogeneous connection graph between modal nodes, the method further includes: The number of co-occurrence locations and the total number of co-occurrences for each co-occurrence pair are used as the spatiotemporal co-occurrence feature.
3. The multimodal spatiotemporal trajectory fusion method according to claim 1, characterized in that, The association strength feature also includes the inter-point mutual information of the corresponding co-occurrence pairs.
4. The multimodal spatiotemporal trajectory fusion method according to claim 1, characterized in that, The determination of spatiotemporally adjacent co-occurrence pairs between the spatiotemporal trajectory data to be processed for each modality, based on preset time and preset spatial thresholds, includes: The preset time threshold and the preset space threshold are dynamically adjusted based on the target movement speed.
5. The multimodal spatiotemporal trajectory fusion method according to claim 1, characterized in that, Before inputting the heterogeneous connectivity graph into the trained graph convolutional neural network to generate the target embedding vector, the method further includes: Acquire multimodal sample spatiotemporal trajectory data and sample relationship pair annotation data; Construct a heterogeneous connection graph of samples based on the spatiotemporal trajectory data of the samples; The heterogeneous connectivity graph of the samples is used as the model input, and the labeled data of the sample relationships is used as the supervision signal. A semi-supervised learning algorithm is used to train the graph convolutional neural network.
6. A multimodal spatiotemporal trajectory fusion device, characterized in that, include: The data acquisition module is used to acquire multimodal spatiotemporal trajectory data to be processed, which includes IMSI data and license plate image data. The relationship pair determination module is used to determine the spatiotemporally adjacent co-occurrence relationship pairs between the spatiotemporal trajectory data to be processed in each modality based on preset time thresholds and preset spatial thresholds. The connection graph construction module is used to establish edges based on the co-occurrence relationship pairs to construct a heterogeneous connection graph between each modality node. Each edge carries attributes including the spatiotemporal co-occurrence characteristics and association strength characteristics of the corresponding co-occurrence relationship pairs. An embedding vector generation module is used to input the heterogeneous connection graph into a trained graph convolutional neural network to generate a target embedding vector. The trajectory fusion module is used to cluster and fuse the target embedding vectors according to a preset similarity threshold to obtain the trajectory fusion result; The association strength feature includes an association strength score; The device further includes: The co-occurrence event fragment statistics module is used to count the co-occurrence event fragments that occur consecutively on the timeline for each co-occurrence relationship pair before establishing edges based on the co-occurrence relationship pairs to construct a heterogeneous connection graph between each modal node; The association strength score determination module is used to determine the association strength score of the corresponding co-occurrence relationship pair based on the number of co-occurrences and the duration of each co-occurrence event fragment.
7. A computer device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal spatiotemporal trajectory fusion method as described in any one of claims 1-5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the multimodal spatiotemporal trajectory fusion method as described in any one of claims 1-5.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal spatiotemporal trajectory fusion method as described in any one of claims 1-5.